Prompt Caching Explained: How to Cut Your AI API Bill

Every major AI API now lets you pay less for tokens you've already sent once before. It's called prompt caching, and for any app that resends a system prompt, a long document, or conversation history on every call, it's often the single biggest lever on the bill — bigger than switching models.

What prompt caching actually does

Most AI apps send more than the user's new message on every request. A chatbot resends the whole conversation so far. A document assistant resends the PDF it's answering questions about. A coding agent resends the same long system prompt and tool definitions on every turn. Without caching, the provider reprocesses all of that repeated text from scratch, at full price, every single time — even though nothing in it changed.

Prompt caching stores the model's internal representation of a chunk of text the first time it's processed, then reuses that stored representation on later calls that start with the exact same text. The provider still has to read the new part of your prompt normally, but the repeated prefix is served from cache at a steep discount. The catch is the match has to be exact and start from the beginning: change one word in your system prompt, or move the document to a different position in the message, and the cache misses.

How the three big providers implement it

The mechanics differ enough between providers that a strategy tuned for one can silently stop working on another.

OpenAI's version requires the least effort: send a prompt over the minimum size (1,024 tokens for most models) with an identical prefix to a recent call, and the discount applies automatically — there's nothing to configure. Anthropic's is the opposite: you have to explicitly mark where the cacheable portion ends, and the very first call that writes the cache costs slightly more than normal, not less, because the provider is doing extra work to store it. That premium only pays off if a later call actually reuses the cache before it expires. Google splits the difference — implicit caching needs no code changes on Gemini 2.5 and newer, while explicit caching gives you a guaranteed, longer-lived cache in exchange for a small storage fee and some setup.

The minimum-length trap

All three providers require the cacheable prefix to clear a minimum token count before caching kicks in at all — commonly in the 1,024–2,048 token range, and higher still for some newer or smaller models. A short system prompt, a two-paragraph set of instructions, or a brief bit of conversation history simply won't qualify, no matter how many times you resend it. This is the most common reason a team tries prompt caching, sees no change in their bill, and concludes it doesn't work: their reused content was never long enough to be cached in the first place. If you want to benefit from caching, it's sometimes worth deliberately front-loading a request with a longer, stable block of instructions or reference material rather than keeping everything terse.

When caching actually pays off

Caching only helps when the same prefix is sent more than once within the cache's lifetime, so the math depends entirely on your traffic pattern:

• Long system prompts sent on every call — a coding agent, customer support bot, or tool-calling app that resends the same multi-thousand-token instructions and tool schemas on every turn is the textbook case. The prefix never changes, so nearly every call after the first is a cache hit.
• Multi-turn conversations — as a chat gets longer, each new turn resends the entire history before it. Caching the growing prefix means turn ten only pays full price for the newest message, not for re-processing the previous nine.
• Repeated document Q&A — asking several questions about the same long PDF or codebase benefits enormously, since the document itself is the cacheable part and the questions are the small, uncached remainder.
• One-off requests — a single question with no reused context gets no benefit at all, and on Anthropic's model, actively costs slightly more due to the cache-write premium if you mark a breakpoint you never revisit.

The other variable is time. A cache that expires after five idle minutes is only useful for traffic dense enough to keep hitting it — a busy production endpoint benefits far more than a script that runs once an hour. That's the practical reason Anthropic's 1-hour cache and OpenAI's longer retention on newer models exist: they trade a higher write cost, or provider discretion, for a wider window in which a hit is likely.

Estimating the real saving

The headline "90% off" numbers describe only the cached portion of a request, not the whole bill. A request with a 4,000-token cached system prompt and a 50-token new question saves close to 90% overall, because almost everything sent was cacheable. A request with the same system prompt but a 3,000-token new question — a long user message or a big chunk of new context — saves far less in percentage terms, because most of the tokens in that call were never eligible for the discount. The saving scales with how much of each request is genuinely repeated, not with the discount rate alone.

To see this in dollar terms for your own usage, run your token counts and monthly request volume through the AI API Cost Calculator, or check how a specific block of text tokenizes with the AI Token Calculator before deciding what's worth caching. If you're comparing models that support caching differently, the AI Model Cheat Sheet lists cached and uncached input prices side by side, and AI Pricing tracks each provider's own published rates as they change.