Start with your own invoice
Every agent platform I have seen has an opinion about prompt caching and no measurement of it. The opinion is usually "we have a static prefix, so we are fine." The measurement is usually a number you have never looked at.
We exported the provider activity for a 30-day window and filtered strictly to our own application name so that other traffic on the same account could not flatter the numbers. That left 397,648 requests. Aggregate prompt volume was 50.6 billion tokens. What we actually paid was $1,799. What cache discounts saved was $10,629.
Read that ratio again. The cache was worth six times the invoice. Caching was not an optimisation on top of the cost model; it was the cost model.
What the blended number hides
The blended cache hit rate was 88.5%. That sounds healthy and it is deeply misleading, because a blended average hides the distribution underneath it. Decomposed by route, the same data looked like this:
| Route | Cache hit rate |
|---|---|
| Primary workhorse route | 96% |
| Provider-bounce era | 46 – 64% |
| One hosted model, one provider | 0.0% |
| Blended | 88.5% |
The 96% is what the architecture is capable of. The 0.0% is not a near-miss, it is a total failure, and one total failure among high-volume routes drags the blended average down by more than the ceiling can recover. An average is the wrong instrument. What you want is the minimum per route, because that is where the money is leaking.
Prompt caching is not a hit rate you observe. It is a property of how you assemble context. If two consecutive requests share a byte-identical prefix, the provider can serve the shared portion from cache. If they do not, it cannot, no matter how similar the requests look to a human.
The four things that break a prefix
1. Anything time-varying near the front
A timestamp, a session id, a "current date" line, a random per-request nonce. If it appears before your large static block, you have invalidated the cache for every request. Content like this belongs at the very end of the prompt or nowhere.
2. A moving tool list
Tool schemas are usually injected near the top because that is where models attend to them best. If your tool set changes shape between turns — because you lazily register capabilities, or because a plugin connected, or because the ordering is non-deterministic — the prefix changes and the cache misses. We treat the static prefix as a contract: same bytes, same order, every turn.
3. Reordering that looks harmless
Map iteration order, directory listings, a set that got serialised, sort instability. Any of these can produce a prompt that is semantically identical and byte-different. It caches or it does not; "semantically identical" is not a thing the provider can see.
4. Provider routing
This is the one nobody owns. The same model name can be served by several backends behind a router, each with its own cache. If your requests bounce between them, or if you fall back to a different model on error, you start every conversation cold. The "provider-bounce era" in the table above is exactly that: 46 to 64 percent, not because the prompts were wrong, but because the traffic was being scattered.
Making the good case the default
The work split into three parts.
- Audit the cache path end to end. Read every line that contributes to the prompt, in order, and mark which parts are static and which are per-turn. Anything per-turn that sits earlier than necessary is a bug.
- Compare against a reference implementation. We read through another agent harness with a well-regarded caching story and diffed our assumptions against theirs. Most of the value was in having something concrete to be wrong relative to.
- Pin the routing, not just the prompt. Identify which model and provider combination gives you the 96% and make that the deliberate default rather than the lucky outcome. Then treat fallback as a degraded mode you can see in telemetry, not an invisible variable.
Turn-start latency came along for free
The bill was the reason to do the work. The latency was the surprise. A cache miss means the provider recomputes the prefix before it can emit the first token, so cache hit rate shows up directly in time-to-first-token. After the fixes, turn-start latency fell by roughly 500 milliseconds — on every turn, for every user, without touching the model or the network.
That is the part I would tell anyone starting this project: the cost win and the experience win are the same fix. You are not trading one for the other.
How to know you actually fixed it
Do not trust a benchmark that runs in a loop against a warm cache. We wrote a headless harness that drives the real runtime with human-shaped prompts — no desktop shell, no synthetic repeated strings — and reports hit rate per route alongside latency and cost. If the harness cannot reproduce a miss, it cannot prove a fix either.
Three rules I would keep:
- Report per-route, and lead with the minimum. Blended averages hide the failures that matter.
- Treat the static prefix as an interface. It has a contract: same bytes, same order, or you broke it.
- Attribute the saving. "We optimised caching" is not a number. "We moved the workhorse route from 46% to 96% and saved $10,629 in 30 days" is.
The ceiling is not 100% and chasing it is a waste. Prompt caching only helps the part of a request that repeats; a genuinely new user question is new tokens and always will be. The goal is to stop losing the part that should have been free.