Every major provider now sells prompt caching, and the headline is good: the repeated part at the front of a request costs a fraction of the normal rate on the second and subsequent calls. Anthropic, OpenAI and Google all price a cache read at roughly a tenth of an input token, give or take.
Teams switch it on, see a cache hit rate near zero, and conclude it does not work for their use case. Usually it works fine and their prefix is not stable.
What caching actually requires
A prompt cache is keyed on an exact prefix. The provider hashes the beginning of your request and looks for a match. The match has to be byte-identical, and it has to be at the front — a cache covers a prefix, not a set of fragments.
So this caches well:
[ system prompt ][ tool definitions ][ company context ][ conversation ]
─────────────── stable, cached ───────────────────── ── varies ──
And this caches never:
[ system prompt ][ retrieved passages ][ tool definitions ][ question ]
── stable ── ── different every turn ── ── now downstream of the change ──
One variable block near the front invalidates everything after it. Tool definitions that were perfectly stable stop being cacheable because something above them moved.
The four things that break it
Retrieval output near the top. This is the big one. A RAG step returns different passages for every question, and the natural place to put them is right after the system prompt, where the model will weight them. That single decision moves your entire tool block and company context behind a boundary that changes every turn.
Timestamps. Today is 30 October 2026, 14:52. A current time in the system prompt means a fresh prefix every minute. If the model needs the date, give it the date and not the time — or move it to the end.
Non-deterministic serialisation. Tool definitions built from a dictionary or a Set can come out in a different order between processes. JSON key order from an unordered map is the classic version of this. The content is identical; the bytes are not; the cache misses and nobody can see why.
Per-user interpolation at the front. You are helping Sam Lee at BigCo. Personalising the first line gives every user their own cache, which for most teams means every cache is cold. The same sentence a few thousand tokens lower costs you nothing.
The arithmetic, roughly
Take a support assistant: 1,800 tokens of system prompt and tool definitions, 6,000 tokens of company context, a 200-token question, and 40,000 requests a month.
With a stable prefix, 7,800 tokens are cached and 200 are fresh. With retrieval inserted at position two, nothing is cached and all 8,000 tokens are charged at full rate on every single call. At a representative $3 per million input tokens against $0.30 per million cached, that is about $960 a month versus roughly $120 — the same answers, an eightfold difference, decided by where one block sits in the array.
Those are illustrative figures and your prices will differ. The ratio is the point, and the ratio is what makes prefix order an architectural decision rather than a tuning detail.
What to do instead
Put the variable part last. Stable system prompt, stable tool definitions, stable company context, then the conversation, then the question. If something must change per request, it goes at the end.
Make the company context genuinely stable. This is where a memory layer differs from a retrieval step in a way that shows up on the invoice. If what the model knows about your business is assembled fresh from a vector search every turn, it cannot be cached by construction. If it is a durable layer the model queries through tools when it needs something, the prefix stops moving. Tool calls happen after the cached prefix and cost only what they return.
Serialise deterministically. Sort keys. Sort tool definitions. Pin the order and write a test that fails if it changes, because nothing else will tell you.
Measure the hit rate, not the bill. Every provider reports cached versus fresh input tokens per request. A bill going up is ambiguous — more usage or worse caching, you cannot tell. The hit rate is not ambiguous. If it is below about 70% on a repetitive workload, something near the front of your prompt is moving.
Watch the TTL. Caches expire, typically in minutes, and some providers charge a small premium to write one. A workload with long gaps between requests can pay the write cost repeatedly and never collect the read discount. Bursty traffic caches beautifully; one request every ten minutes may cost more cached than not.
The connection to context
Caching is a partial remedy for a problem better solved upstream. If you are re-sending six thousand tokens of company background on every turn, the cache makes it cheap but does not make it necessary — and it is still six thousand tokens of the model's attention spent on material that was mostly irrelevant to this particular question.
We have written separately about what it costs to paste your company into every prompt. Caching lowers the unit price of that habit. Not having the habit lowers it further, and tends to improve the answers, because a model asked to find one fact among six thousand tokens of preamble is doing a harder job than one that can fetch the fact.
Switch caching on regardless — it is nearly free and the gain is real. Just measure the hit rate afterwards, because an unmeasured cache has a way of being a cache in name only.