Ask a finance team what they spend on AI and they will read you the number at the bottom of the provider invoice. Ask what the money bought and the room goes quiet, because the invoice is denominated in tokens and nothing in the business is.
The line is worth decomposing, because one component of it is much larger than people expect and is almost entirely avoidable.
The invoice has four parts
The question. Tiny. A sentence.
The answer. Small, and the part everyone pictures when they imagine the bill.
The background. The ticket thread somebody pasted in. The two-page policy document. The CRM record. The twelve rows of SQL output an agent fetched on step three. This is usually the bulk of it.
That same background again, and again. Here is the part that surprises people. In a conversation, every turn re-sends everything that came before. Paste a long thread on turn one and you pay for it on turn one, and again on turn two, and again on turn six. A ten-turn conversation around one pasted document pays for that document something close to ten times.
Input tokens are cheaper than output tokens, which is why this hides. They are also five to fifty times more numerous in any workload where the model needs to know something about your business. On a per-request basis the background usually dominates.
Why the obvious fix does not work
The obvious fix is "paste less". It fails for a simple reason: the person pasting does not know which parts matter. They paste the whole thread because trimming it means reading it carefully first, and reading it carefully was the job they were trying to hand over.
The second fix is a bigger context window, which is not a fix at all. A million-token window does not make tokens free; it removes the pressure to be selective while keeping the price per token exactly where it was. Larger windows have made this bill bigger, not smaller.
What retrieval actually changes
The alternative is that nobody pastes anything, because the assistant can fetch what it needs.
Not the whole ticket — the four facts from the ticket that bear on the question. Not the CRM record — the renewal date and the owner. A compact set of statements, assembled at the moment of asking from a graph of what the company knows, and carried into the request instead of the documents they were extracted from.
The size difference is the point. A thread is thousands of tokens. The three facts that matter from that thread are a few dozen. And because the facts persist, the second question about the same account does not re-fetch anything — it is already known.
This is not free, and it is worth being precise about what it costs. The retrieved facts are themselves tokens. Extracting durable facts from each exchange is a small model call. Embeddings cost something, and so does the consolidation pass that merges near-duplicates. None of that is nothing. It is simply much smaller than re-sending a document ten times, and it is paid once per fact rather than once per turn.
Doing the arithmetic on your own numbers
Four figures, and you can get three of them from your provider dashboard this afternoon:
- Your monthly provider spend.
- Roughly what share of it is background context rather than the question and the answer. If your usage dashboard breaks out input and output tokens, you can read this directly rather than guessing.
- How much of that background a few retrieved facts could stand in for.
- What share of your turns are genuinely easy — lookups, short questions, acknowledgements — because those do not need your most expensive model at all.
The fourth one compounds with the first three, and it is the one teams under-estimate most. A question like what is Sam's email is a lookup. If it is being answered by the same model that writes your migration plans, you are paying frontier prices for a database query, every time anyone asks.
Put your figures into the calculator and it will show you the two mechanisms separately: what shrinking the context saves, and what routing the easy turns down a rung saves on what is left.
The number nobody has
The reason this analysis is rare is not that it is hard. It is that most teams cannot see the inputs. The provider invoice is one number for the whole organisation. It does not say which team, which person, which model, or how much of each request was background.
So the first thing worth doing is not an optimisation at all. It is getting a usage breakdown by user, by team and by model, with the token counts per request. Once that exists, the two large savings above stop being an argument and become a measurement — and you will probably find that a handful of workflows account for most of the bill, which makes the next decision considerably easier.