Prompt Caching: How It Works and What It Actually Saves

Prompt caching lets a model skip work it has already done. When two requests begin with the same text — the same system prompt, the same document, the same conversation so far — the provider can reuse the internal state it computed the first time instead of processing those tokens again. You pay a fraction of the normal price for the reused part, and the answer starts sooner.

The savings are not marginal. On a 50,000-token prefix reused across 20 requests, the measured discount ranges from 71% to 93% of the input bill depending on the provider, using the prices on our LLM API pricing page. The feature is also the single easiest cost reduction available to most applications, because on several providers it is already on and requires no code at all.

This guide explains the mechanism, shows it working on a local model where you can watch the clock, compares what the four major providers charge and restrict, and works through when caching does not pay off.

The mechanism, briefly

To generate text, a model converts every input token into internal vectors called key-value states, or the KV cache. These states are what let the model refer back to earlier tokens. Computing them for a long prompt is the expensive part of “reading” your input.

OpenAI’s documentation describes the trick plainly: the cache “preserves that state for a reusable prefix: the unchanged tokens at the beginning of a prompt.” A later request with the same opening can load those states instead of recomputing them.

Two consequences follow, and between them they explain almost every question people have about the feature:

1. It is a prefix cache, not a content cache. Matching starts at the first token and stops at the first difference. Reordering your prompt so the document comes second does not “still match” — it matches nothing.
2. Only the input side benefits. Generation is unaffected. Anthropic states it directly: “Prompt caching has no effect on output token generation. The response you receive is identical to what you would get if prompt caching were not used.”

Watching it happen

You do not need an API key to see this. Ollama, running a model locally, does prefix caching too, and reports how long it spent processing the prompt.

The test: a 4,590-token prompt — an instruction plus a repeated financial report — sent to gemma3:270m on a CPU-only laptop, four times.

Run	Prompt	Tokens processed	Prompt processing
1	Cold, nothing cached	4,590	19.01 s
2	Same prefix, different question at the end	4,591	0.14 s
3	Two words changed at the start of the prefix	4,591	19.98 s
4	Back to the original prefix, third question	4,590	0.15 s

Run 2 is 136 times faster than run 1 for the same amount of text. Nothing was compressed and no shortcut was taken on quality — the model simply loaded states it had already computed.

Run 3 is the lesson that costs people money. The only change was financial analyst → financial auditor, twelve characters, at the very beginning. Every one of the 4,591 tokens had to be processed again, and the request took as long as the cold one.

The script is 50 lines and runs against a local Ollama install:

import json, time, urllib.request

PREFIX = "You are a financial analyst. Here is the report:\n\n" + REPORT * 60

def ask(prompt, label):
    payload = {"model": "gemma3:270m", "prompt": prompt, "stream": False,
               "options": {"temperature": 0, "num_predict": 12, "num_ctx": 8192}}
    req = urllib.request.Request("http://127.0.0.1:11434/api/generate",
                                 data=json.dumps(payload).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        d = json.loads(r.read())
    print(label, d["prompt_eval_count"], round(d["prompt_eval_duration"] / 1e9, 2), "s")

ask(PREFIX + "\n\nQuestion: what was gross margin?", "cold")
ask(PREFIX + "\n\nQuestion: what was free cash flow?", "same prefix")

A hosted API does the same thing across requests from your account, with a price attached to each half of the deal.

What it costs: the two rates

Providers split cached tokens into two prices:

• Cache write — putting a prefix into the cache the first time. Sometimes it costs more than a normal input token, sometimes nothing extra.
• Cache read — reusing it. Always much cheaper than the normal input rate.

Anthropic’s multipliers are the clearest statement of the pattern: 5-minute cache writes cost 1.25× the base input price, 1-hour writes 2×, and cache reads 0.1× — with Claude Fable 5.1 and Claude Mythos 5.1 reading at 0.025× instead.

So the question is when the write premium pays for itself. Writing a prefix and reusing it once costs 1.25 + 0.1 = 1.35× the base rate, against 2× for processing it twice uncached. The second request already comes out ahead. OpenAI’s documentation makes the same calculation and extends it: “across ten requests, one write and nine full reads cost 2.15×, compared with 10× without caching.”

Precisely, with a write multiplier w and a read multiplier r, caching wins once the prefix is used more than (w − r) / (1 − r) times:

Write premium	Read rate	Pays off after
1.25× (Anthropic 5-minute, OpenAI GPT-5.6+)	0.1×	1.28 uses
2× (Anthropic 1-hour)	0.1×	2.11 uses
None (Gemini, DeepSeek, OpenAI before GPT-5.6)	0.02×–0.1×	immediately

In other words: if a prefix is ever reused even once, caching is already cheaper. There is no realistic case where a prefix you genuinely repeat is not worth caching.

What it saves, on real prices

Here is the same scenario priced against every model on our pricing page that publishes a cached rate: a 50,000-token prefix — a system prompt plus a document — held constant across 20 requests. Cache writes are charged where the vendor charges for them.

Model	Input $/M	Cache write	Cache read	20 requests, no cache	With caching	Saved
Claude Fable 5.1	$10.00	$12.50	$0.25	$10.00	$0.86	91.4%
Claude Opus 5	$5.00	$6.25	$0.50	$5.00	$0.79	84.2%
Claude Sonnet 5	$2.00	$2.50	$0.20	$2.00	$0.32	84.2%
Claude Haiku 4.5	$1.00	$1.25	$0.10	$1.00	$0.16	84.2%
GPT-5.6 Sol	$4.00	$5.00	$0.40	$4.00	$0.63	84.2%
GPT-5.6 Terra	$2.00	$2.50	$0.20	$2.00	$0.32	84.2%
GPT-5.5	$5.00	none	$0.50	$5.00	$0.72	85.5%
Gemini 3.8 Flash	$0.75	none	$0.075	$0.75	$0.11	85.5%
DeepSeek V4 Pro	$0.66	none	$0.022	$0.66	$0.05	91.8%
DeepSeek V4.1 Flash	$0.15	none	$0.003	$0.15	$0.01	93.1%
Grok 4.6	$2.00	none	$0.50	$2.00	$0.57	71.2%
Grok 4.5	$2.00	none	$0.30	$2.00	$0.39	80.8%

Prices read from our LLM API pricing data on September 22, 2026; output tokens are excluded because caching does not touch them.

Three things stand out.

The spread between providers is real. DeepSeek reads cached tokens at 2% of the input rate and xAI at 15–25%. On identical usage that is 93% saved against 71% — the difference between a bill cut to a fourteenth and one cut to a third.

The write premium barely matters at scale. Claude Fable 5.1 charges 1.25× to write and still lands at 91.4%, because one write is amortised over nineteen reads. Worry about the read rate, not the write rate.

These are input savings only. If your application generates long answers from short prompts, caching will do very little. It is a tool for prompt-heavy workloads: long system instructions, documents, few-shot examples, tool definitions, growing conversations.

The four providers, side by side

	Anthropic	OpenAI	Google Gemini	DeepSeek
On by default	Automatic caching, plus explicit breakpoints	Yes, implicit; explicit on GPT-5.6+	Yes, implicit	Yes
Explicit control	cache_control breakpoints	prompt_cache_breakpoint (GPT-5.6+)	Separate cache objects via generateContent	None
Minimum prefix	512–4,096 tokens, varies by model	1,024 tokens (GPT-5.6+)	2,048–4,096 tokens	Not documented
Lifetime	5 minutes, or 1 hour at 2× write	30 minutes (GPT-5.6+); up to 24 h on earlier models	TTL you set on explicit caches	Hours to days, best-effort
Write charge	1.25× (5m) or 2× (1h)	1.25× on GPT-5.6+, none earlier	None for implicit	None
Read charge	0.1× (0.025× on Fable/Mythos 5.1)	0.1× on GPT-5.6+	~0.1×	0.02×–0.033×
Storage fee	No	No	Yes, per hour, on explicit caches	No
Usage fields	cache_creation_input_tokens, cache_read_input_tokens	cached_tokens	usage.total_cached_tokens	prompt_cache_hit_tokens, prompt_cache_miss_tokens

Four details in that table deserve expanding, because each one catches people out.

Minimum prefix lengths are not uniform

A prefix below the minimum is simply not cached, silently. Anthropic publishes the thresholds per model: 512 tokens for Claude Fable 5.1, Mythos 5.1, Opus 5, Fable 5 and Mythos 5; 1,024 for Opus 4.8, Sonnet 5 and the Sonnet 4.x line; 2,048 for Mythos Preview and Opus 4.7; 4,096 for Opus 4.6, Opus 4.5 and Haiku 4.5.

Note how that runs against intuition: the cheap, fast Haiku 4.5 needs a 4,096-token prefix, eight times what the flagship needs. Gemini is similar — 4,096 tokens for the Gemini 3.x Flash models and 3.1 Pro Preview, 2,048 for Gemini 2.5 Flash and Pro. OpenAI requires 1,024 visible input tokens on GPT-5.6 and later, and says that on earlier models the minimum “varies by request settings, including tools, images, output schemas, reasoning effort, and verbosity.”

If your system prompt is 600 tokens, caching may do nothing at all on the model you picked.

Google charges rent

Anthropic and OpenAI charge for writing a cache entry and then let it expire. Google’s explicit caches charge storage by the hour: the pricing page lists $0.50 to $1.80 per million tokens per hour depending on the model, alongside the discounted read price.

That changes the arithmetic. Holding a 50,000-token cache for an eight-hour working day at $1.00 per million per hour costs $0.40 in storage — more than the entire uncached input cost of the twenty-request scenario above on several models. Explicit Gemini caches pay off for high-frequency reuse of a large context, and lose money when idle. Implicit caching has no storage fee, and for most applications it is the right default.

Lifetimes are short, and the clock may start earlier than you think

Anthropic’s default is five minutes, refreshed free on every hit, so a steadily-used prefix stays alive indefinitely at no extra cost. The one-hour option exists for gaps between five minutes and an hour.

The subtlety is in this line of Anthropic’s documentation: the lifetime “is measured from the start of the request that writes or reads the cache entry, not from the end of its response. Time spent generating a response counts against the lifetime: if a response takes 4 minutes to stream, a follow-up request that reuses the same cached prefix must start within about 1 minute of that response completing.”

For a long agentic turn, the five-minute window can be mostly consumed by the response itself.

Cache hits do not count against your rate limit

A detail worth knowing when you are throttled rather than broke: Anthropic lists improved rate-limit utilisation as a reason to use the one-hour cache, “because cache hits are not deducted against your rate limit.” Caching can therefore buy throughput, not only money.

Structuring a prompt so it caches

The rule follows from the mechanism: stable content first, variable content last.

Anthropic’s cache follows the hierarchy tools → system → messages, and a change at any level invalidates that level and everything after it. So:

[ tool definitions        ]  ← change rarely, put first
[ system prompt           ]
[ documents, examples     ]  ← the expensive, stable part
--------------------------- cache breakpoint
[ conversation history    ]
[ the user's new question ]  ← changes every request

Things that quietly break the match:

• Any timestamp, request ID or “today’s date” near the top. A date string at the start of a system prompt destroys the cache once a day at best, and on every request at worst.
• Reordering tool definitions. OpenAI lists tool “names, descriptions, schemas, ordering” as prefix-affecting. A tool list built from an unordered dictionary can shuffle between runs.
• Toggling a feature. On Anthropic, enabling or disabling web search or citations edits the system prompt; changing the speed setting invalidates system and message caches.
• Changing reasoning effort or thinking budget. Both are rendered into the prompt. OpenAI and Anthropic each now offer a way to change effort without rewriting the prefix — worth using if your app varies effort per request.
• Adding or removing an image anywhere. It invalidates the message cache on Anthropic.
• Switching models. Caches are per model. A fallback to a different model starts from nothing.

And a non-obvious one: caches are isolated per organisation, and on the Claude API also per workspace. Anthropic flags that Bedrock and Google Cloud use organisation-level isolation instead, so the same application can have a different hit rate depending on where it runs.

Why agents benefit most

A chat or agent loop re-sends the entire conversation on every turn. That is the shape caching was built for, and the reason it matters more there than anywhere else.

Take a 10,000-token system prompt and tool set, and a conversation that grows by 1,500 tokens per turn. Across ten turns, the uncached input adds up to the sum of every prefix re-read from scratch:

Turn	Prompt size	Uncached, cumulative	Written to cache	Read from cache
1	11,500	11,500	11,500	0
3	14,500	39,000	14,500	24,500
5	17,500	72,500	17,500	55,000
10	25,000	182,500	25,000	157,500

By turn ten the application has sent 182,500 input tokens for a conversation that only ever contained 25,000 — the same text re-read again and again. With caching, only 25,000 of those tokens are ever written; the other 157,500 are reads.

Priced at a 1.25× write and a 0.1× read, that is the equivalent of 47,000 tokens against 182,500 — the bill drops to about a quarter, and the gap widens with every turn. Where there is no write premium it falls to 22%.

Two practical points for agent loops specifically.

The growing tail needs its own breakpoint. The system prompt and tools are stable, but the conversation is not: each turn appends to it. Providers handle this with a breakpoint that moves. Anthropic’s automatic caching places one at the end of the last block; OpenAI’s implicit mode places one “at the end of the latest eligible message”. If you are placing breakpoints by hand, put one after the stable header and one at the end of the history, so each turn extends the cache rather than rewriting it.

Tool results count. On Anthropic, tool use and tool results can be cached like any other content block, which matters for a loop that calls three tools per turn. OpenAI treats the last tool response in a consecutive group as an eligible breakpoint.

This is also why an agent that changes its tool list mid-run pays twice: modifying tool definitions invalidates the tools cache, and with it the system and message caches behind it. Append-only tool updates are worth the discipline.

Reading the numbers back

Every provider reports what actually happened. Check it before believing any estimate, including the one above.

Anthropic returns three fields, and the third is routinely misread:

"usage": {
  "input_tokens": 50,
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 100000,
  "output_tokens": 503
}

input_tokens counts only the tokens after your last cache breakpoint, not the whole prompt. Total input is the sum of all three — 100,050 tokens here. A dashboard that graphs input_tokens alone will show a cliff the day you enable caching and hide what you are actually sending.

OpenAI reports cached_tokens, rounded down to a multiple of 128 on models before GPT-5.6 and exact on GPT-5.6 and later. Gemini reports usage.total_cached_tokens. DeepSeek splits the input into prompt_cache_hit_tokens and prompt_cache_miss_tokens, which is the easiest format to turn into a hit rate.

A hit rate well below expectations almost always means one of two things: your prefix is under the minimum length, or something near the top of it changes between requests. The local experiment above is a cheap way to build intuition for the second.

When caching does not help

Honest limits, since most write-ups skip them:

• Short prompts. Below the model’s minimum, nothing is cached and nothing is saved.
• Prompts that never repeat. One-off classification of unrelated documents has no prefix to reuse. Cache writes on GPT-5.6+ or Anthropic would make it 25% more expensive.
• Output-heavy work. Generating a long article from a two-line brief is dominated by output tokens, which caching does not touch.
• Long gaps between requests. A prefix used twice an hour will expire between uses on default settings. Either pay for the longer TTL or accept the misses.
• High-variance prompt assembly. If your framework injects anything variable near the top — retrieved chunks before the system prompt, a shuffled tool list, a session ID — the cache will miss constantly while your code looks correct.

There is also a case where caching competes with a cheaper option. Both Anthropic and OpenAI offer a batch API at around 50% off for work that tolerates delay. Anthropic notes the caching multipliers “stack with other pricing modifiers such as the Batch API discount”, so for offline jobs the two combine rather than compete — but if your workload is offline and non-repetitive, batching is the lever, not caching.

What to do this week

In rough order of effort:

1. Check whether you are already getting hits. Log the cache fields for a day. On Gemini, DeepSeek and OpenAI, caching is on by default and you may be benefiting without knowing.
2. Move the variable parts to the end. Timestamps, session IDs and the user’s question belong after everything stable. This is usually a small change and often the whole fix.
3. Measure your prefix. If it is under the model’s minimum — 4,096 tokens on Haiku 4.5 and the Gemini Flash line — decide whether to consolidate context or accept no caching.
4. Add explicit breakpoints if your provider supports them and your prompt has parts that change at different rates. Anthropic allows several; OpenAI allows up to four cache writes per request.
5. Only then consider a longer TTL. It costs 2× to write on Anthropic and is worth it only for gaps longer than five minutes.

Then put the numbers in a spreadsheet. With the read rate, the write rate and your real hit rate, the monthly saving is arithmetic — and our LLM API cost calculator already models a cached share of input if you want to try the scenarios before writing code.

Frequently asked questions

Does prompt caching change the model’s answers?

No. Anthropic states that caching “has no effect on output token generation” and the response is identical to an uncached one. Cached or not, the model sees the same tokens; only the work of processing them is reused.

Is prompt caching enabled by default?

On Gemini, DeepSeek and OpenAI, yes. Anthropic now has automatic caching as well as explicit cache_control breakpoints. Explicit control still matters when different parts of your prompt change at different rates.

How much does prompt caching save?

On a 50,000-token prefix reused 20 times, between 71% and 93% of the input cost depending on the provider, from the prices on our pricing page. The savings apply to input tokens only.

Why is my cache hit rate so low?

Usually either the prefix is shorter than the model’s minimum — up to 4,096 tokens on some models — or something changes near the top of it. Timestamps, session identifiers, reordered tool definitions and toggled features are the usual culprits.

Is cached content private?

Caches are isolated between organisations, and on the Claude API also between workspaces within an organisation. Anthropic describes prompt caching as zero-data-retention eligible: KV representations and hashes are held in memory only and not stored at rest. OpenAI notes it may store encrypted key/value tensors in GPU-local storage as application state.

Does caching help with rate limits?

On Anthropic, yes: cache hits are not deducted against your rate limit, which is a separate reason to use it when you are throughput-constrained rather than cost-constrained.

Related

• LLM API pricing with cached input rates per model, and the LLM API cost calculator
• How to run an LLM locally with Ollama, where you can watch prefix caching directly
• What is RAG? — retrieved chunks belong after the stable prefix, not before it
• Token counter to measure whether your prefix clears the minimum
• Inference, tokens and tokenization in the glossary