Your Cache Is Part of the Model Comparison

A warm cache lets a model reuse earlier processing of matching input. Prompt caching means saving that work so a later request can reuse it. The prefix is the beginning of the prompt: often instructions, tool definitions and shared reference material. Cold means the reusable entry is absent; warm means a matching entry is available.

The details belong to the provider. For example, Anthropic's prompt-caching documentation describes separate charges for writing cached input and reading it back. It also explains that changing content before a cache breakpoint changes the prefix being matched. A discount depends on what your application actually reuses.

That makes a repeated benchmark a particular operating condition. If your experiment sends the same setup repeatedly, it may reward reuse that a queue of unrelated jobs will never achieve. The result can be accurate and still answer the wrong buying question. Ask whether the production workload will keep that prefix stable and send work often enough to reuse it.

Give each candidate the same jobs and acceptance standard, then measure it under deliberately different cache conditions. Use the same repository snapshot, permitted tools and task instructions. Record any provider-specific adaptations. Otherwise, a context change can quietly become part of the apparent model improvement.

Start with a cold run. Then repeat representative work with reusable context available. Finally, introduce the gaps and context changes your production queue actually contains. Check the provider's reported cache usage instead of calling a run warm because you expected a hit. Record input reads, writes, uncached input, output, elapsed time and failed attempts alongside the result.

Keep the rejected jobs in the bill. A route that caches efficiently but needs another builder to finish its work has consumed both routes. Our argument in The Price Card Is Not the Bill still applies: accepted work is the unit you buy. Cache accounting explains part of that cost; it cannot certify the result.

The fallback has to earn its place using the context and cache state it will actually receive. Do not assume warming the first route warms the next one. Test the handoff. Include the failed attempt, the transfer of task state, the receiving model's input processing and the independent check that decides whether the job is finally done.

Picture a routine dependency update that turns into an ambiguous compatibility problem. This is a test scenario, not a customer story. Your inexpensive route reaches its stopping condition, and a stronger model takes over. If the cost estimate counted only the inexpensive route's successful warm runs, it never priced the job that needed help.

That does not make fallback a mistake. It makes escalation part of the route you are evaluating. As we explain in A Cheaper Model Is a Routing Signal, the cheap lane needs an exit. Now prove what that exit costs and whether the receiving worker can finish without asking a person to reconstruct the whole task.

Keep shared instructions and tool definitions stable when they remain correct, and make changing task material explicit. Anthropic recommends placing reusable material before changing content. That is a useful implementation choice after you have decided what context the task requires.

Never preserve stale context to protect a discount. A changed security rule belongs in the next request even if it breaks reuse. A newer specification outranks yesterday's cached instructions. Give the context bundle a recorded version and connect it to the evidence for each result, so you can tell whether a cheaper run used different rules.

Gunner Technology's position is that a factory should spend more to apply the right instruction before it spends less to repeat the wrong one. People decide which constraints matter and judge whether the work meets them. Assembling the approved context, recording usage and routing a failed attempt are repeatable jobs the machinery should handle.

Approve a candidate for the workload it passed. If its advantage depends on frequent jobs sharing a stable prefix, send it that class of work. Test sporadic jobs separately. Keep a path for cold starts and failures, with a spending limit and a stopping condition attached.

Before switching production traffic, write down what would overturn the decision: lower reuse, more rejected work, slower completion or repeated escalation. Then make those signals change the route. A dashboard that reports the lost saving after the month closes is too late to govern the next attempt.

Start with a repeatable queue in a repository you own. Compare warm, cold and fallback behavior against the same proof. Move the work when the complete route earns it. The useful saving is the one your factory can reproduce on the next real job.