Context windows · Why it’s finite

The bill

Why not just make it enormous?

Inside a model, every token looks at every other token to work out what it means in this particular sentence. For n tokens that’s n × n comparisons, so the work grows with the square of the length, not in step with it. Memory grows too: the model keeps a running cache for every token it has seen so far.