Gains:
- Explain the prefix matching logic of prompt caching
- Increases cache hit by putting fixed context first and variable context after
- Can calculate cache write/read economics and break-even point
An LLM product looks cheap in prototype; When you step up to the scale, the bill surprises. In most workloads, most of the bill comes from the same fixed context that is sent over and over again with each request: a long system prompt, a rulebook, reference documentation. Prompt caching eliminates exactly this waste. In this unit, you will learn how the cache works, how to arrange the prompt to hit, and how to calculate the break-even point of the cache economy. When installed correctly, it alone can cut your bill in half or even lower.
How Does Cache Work? The One Immutable Rule
Prompt caching is a prefix match. The provider temporarily stores the tokens it has processed since the beginning of your prompt. If the prompt starts with the same prefix on the next request, this common part is not recalculated; It is much cheaper to read than cache.
One immutable rule follows from this: If a single byte changes anywhere in the prefix, the entire cache becomes invalid from that point onwards. That is, fixed content should be at the beginning and variable content should be at the end. If you put a line at the beginning of the system prompt that changes with each request, such as "Today's date: 18.07.2026", everything behind it will not be able to enter the cache.
The processing order is usually: tools → system prompt → messages. You put the cache point (breakpoint) at the end of the fixed section.
Cache Economy
Cache has three price tiers:
- Cache write: Storing for the first time. ~1.25x normal input price (for 5 minute storage).
- Cache read: Reading on subsequent requests. ~0.1 times the normal input price — that is, one tenth.
- Normal input: The part that does not enter the cache and is processed at full cost each time.
Break-even point: First request pays write premium (1.25×). From the second request, the reading (0.1×) comes into play. Roughly, you'll be neck and neck on two requests; After that, it is net savings. The larger the fixed context and the more requests it is reused, the larger the gain becomes.
Scenario
Does cache work?
Large fixed system prompt, thousands of requests
Yes — highest earnings
Many questions on the same reference docs
Yes
Completely different short text for each request
No — write bonus is wasted
One time request
No — no reading at all
Date/ID changing with each request at the system prompt
No — prefix is broken, hit is zero
Step by Step: How to Set Up a Hit Prompt?
- Separate constant and variable. What content never changes (system prompt, rulebook, documentation)? Which changes with each request (user question, date, ID)?
- Put the constant at the beginning. During processing, the part that comes first (tools, system) must be stable.
- Put the variable at the end. User's current question, last.
- Place the sign at the end of the border. Put the cache point in the last block of the fixed part.
- Verify hit. Check if cache_read_input_tokens is greater than zero in the usage field in the response. If zero, there is a hidden disruptor in the prefix.
{ "system": [ { "type": "text", "text": "{{large_constant_system_promptu_and_rules}}", "cache_control": { "type": "ephemeral" } } ], "messages": [ { "role": "user", "content": "{{user_current_question}}" } ]}
Tip: Don't guess cache hits, measure them. If usage.cache_read_input_tokens is still zero on consecutive requests, a silent breaker (datetime.now() at system prompt, unordered JSON, list of tools changing with each request) is running. Compare the raw prompt of the two requests byte by byte and find the difference.
Silent Disruptors
Typical patterns that unknowingly corrupt the cache:
# BREAKER: embedding information in the system prompt that changes with each request "Today's date: {{now}}. You are an assistant..." ← prefix changes with each request, hit is zero# TRUE: move the variable to the messagesystem: "You are an assistant..." ← constant enters the cachemessages: [{role: user, content: "Today is {{now}}. Question: ..."}] ← variable at the end
Other breakers: JSON sorted differently on each request (keep keys in fixed order), list of tools varying by user (tools are processed first; nothing goes into the cache if they change), changing the model mid-conversation (caches are model specific).
Weak prompt / Strong prompt (cache friendly structure)
# WEAK (cache busting build)system: "Date: 18.07.2026 14:32. User: Ahmet (id 8842). You are a support bot. Rules: ...(2000 tokens)..."
# STRONG (cache-friendly structure)system: "You are a support bot. Rules: ...(2000 tokens, never changes)..." [cache sign]messages: [ { role: user, content: "Date: 18.07.2026 14:32. User id: 8842. Question: how do I initiate my refund?" }]
In the weak version, the rule block of 2000 tokens is processed at full cost on each request. In the strong version, the same block is written once and read on all subsequent requests for a tenth of the price.
Three Mini Cases
Case 1 — Caching the rulebook. An accounting automation was adding the 12,000 token rulebook to each invoice; 5,000 requests per day. Cacheless input costs ~$180 per day. They kept the rulebook constant and cached it: first requests paid a write premium, subsequent reads 0.1×. Input cost dropped ~90% to ~$18 per day.
Case 2 — Cost of hidden date line. One team set up a cache but were getting no hits; cache_read_input_tokens was always zero. Reason: There was datetime.now() in the first line of the system prompt, the prefix was changing with each request. When we moved the date to the user message, the hit rate suddenly increased from 0% to 94%.
Case 3 — Misplaced cache. A search application was sending completely different short queries with each request; They eagerly added a cache sign. With no common prefix, each request paid only a write premium, no reads — increasing the cost. They removed the sign. Lesson: cache pays only if there is a large and constant prefix that is reused.
Common mistakes
- Mixing constant and variable: When the variable content is in the prefix, the hit is reset.
- Embedding date/ID in system prompt: The most common silent disruptor.
- Not measuring the hit: If cache_read_input_tokens is not checked, waste will not be noticed.
- Adding cache when there is no public prefix: You only pay the write premium, the cost increases.
- Changing the vehicle list or model: The prefix is broken from the beginning; everything is rewritten.
- Forgetting the minimum cache size: Very short caches (under ~1–4k tokens depending on the model) will not enter the cache silently.
Deeper: Designing Cache by Workload Type
The actual payoff of caching varies depending on the nature of your workload; so get to know your traffic first. Three typical patterns and correct installation:
Common system prompt, different questions. Most common enterprise pattern: a large system prompt (role, rules, maybe reference document) with hundreds of different user questions. Here the fixed part (system) is cached initially; each new question pays full price only for its own small portion. The gain is very high because the large portion is recited repeatedly at a tenth of the price.
Multi-round monologue. As a conversation drags on, each new round builds on top of all previous history. If you put the cache flag at the end of the last round, each request reuses the previous conversation prefix; hits accumulate as the conversation grows. This dramatically reins in the cost of long assistant sessions.
The shared prefix is the last bit to change. Multiple requests share a large set of fixed priors (sample set, instructions) but are separated by a single question at the end. You put the cache pointer at the end of the shared part; Otherwise, each request would write its own separate cache and none of it would be read.
One caveat: the cache depends on the model and a certain minimum size. Very small prefixes (under a few thousand tokens, depending on the model) will not silently enter the cache even if you flag them — cache_creation_input_tokens remains zero. Also, changing the model mid-conversation invalidates the entire cache; If a different task requires a cheap model, keep the main flow in one model and put the side job in a separate call.
In summary
Prompt caching is a prefix match: fixed content should be at the beginning, variable content should be at the end. For a large, reused context, the read cost is a tenth of the full price, roughly breaking even in two requests. The most common mistake is to corrupt the prefix by embedding variable data in the system prompt; You verify the hit by measuring it in the usage field.
Application task
Choose a workload. (1) Divide the content into two columns: "never changes" and "changes with every request". (2) Redraw the prompt structure, putting the constant part at the beginning and the variable part at the end. (3) Estimate the token size of the fixed part and compare the monthly cost with/without cache. (4) Note which field (cache_read_input_tokens) you will verify the hit from.
checklist
- [ ] I can explain that cache is prefix matching and the only immutable rule.
- [ ] I can increase the accuracy by putting the fixed content at the beginning and the variable at the end.
- [ ] I know write/read economics and the two-request break-even point.
- [ ] I can recognize silent disruptors (date, unordered JSON, changing vehicle list).
- [ ] I can verify the hit with usage.cache_read_input_tokens.