What is a context window?

A context window is the maximum amount of text a model can hold in a single request, measured in tokens. It covers everything: the system prompt, tool definitions, any documents or records supplied, the conversation so far, and the answer the model is about to produce. Input and output share the budget.

A token is roughly four characters of English, so a thousand tokens is about 750 words. Current frontier models offer windows from a couple of hundred thousand tokens to a million or more.

What a bigger window does not fix

The obvious reading is that a large window removes the need to choose what to send. Paste the handbook, paste the ledger, let the model sort it out.

Three things get in the way.

Attention is not uniform. Models reliably recall material at the beginning and end of a long input and are measurably worse in the middle — the effect usually called "lost in the middle". Filling a million-token window does not give you a million tokens of equally reliable reading. A fact buried at 60% depth may as well be absent.

You pay for all of it, every turn. A conversation is stateless underneath; each turn re-sends the whole history. Twenty turns over a 50,000-token context is a million tokens charged, for one conversation. This is the arithmetic behind what it costs to paste your company into every prompt, and it is the largest line on most AI bills.

Latency scales with input. Time to first token grows with how much there is to read. A window stuffed to capacity makes an interactive assistant feel broken even when the answer is right.

So the practical limit is not the window size. It is the point past which more context makes answers worse, slower and dearer — and that point arrives long before the model's advertised maximum.

What the space is actually for

Treat the window as working memory for one question, not as storage.

The stable material — instructions, tool definitions, a small amount of durable company context — belongs at the front, where it can be cached. Everything variable belongs at the end. Getting that order wrong invalidates prompt caching for everything downstream of the first thing that moves, which is the most common reason a cache hit rate sits near zero.

Everything else should be fetched rather than pre-loaded. A model that can call a tool when it discovers it needs a figure does not need the figure pre-emptively pasted, and the tool result arrives after the cached prefix, so it costs only what it returns. This is the difference between a system that pushes context and one that lets the model pull it.

Window, memory and history

These are three separate things and conflating them causes most of the confusion:

• The context window is how much the model can read in one request. It is gone when the request ends.
• Agent memory is what persists between requests, so the next conversation does not start from nothing.
• Conversation history is the transcript of the current exchange, which occupies the window and grows until something truncates or summarises it.

A larger window extends the third. Only memory addresses the first problem anyone actually has, which is that the model does not know anything about your company until something tells it — and would rather not be told again every single turn.

Counting it

Every provider exposes a tokeniser and reports input, output and cached token counts per request. Use the real counts rather than estimating from characters; code, JSON and non-English text tokenise far less efficiently than prose, sometimes by a factor of two or three. A context budget built on a four-characters-per-token rule of thumb will be wrong exactly where it matters, which is on the structured data you are pasting in.