Prompt caching

What it is

Prompt caching works on the prefix of a request: the opening block of tokens that stays identical across many calls. The provider keeps a representation of that prefix for a limited period and matches incoming requests against it. Cached input tokens are typically billed at a lower rate than fresh input tokens, and cache entries expire after a short window of inactivity.

Why it matters

Retrieval-based and agentic applications often send the same large context repeatedly, so the same tokens get paid for and processed hundreds of times a day. Caching cuts that duplication, which makes long system prompts, extensive brand guidelines and large document contexts economically viable. Faster responses also matter for chat interfaces and for pipelines where several model calls run in sequence.

How it works

Practitioners restructure prompts so everything stable sits at the front and everything variable sits at the end: system instructions, tools and reference documents first, user question last. They keep the prefix byte-identical, avoiding timestamps, session IDs or reordered content that would break the match. Providers differ in whether caching is automatic or requires explicit cache markers, so implementation follows the specific API documentation.

When it applies

It applies whenever the same substantial context is sent repeatedly within a short window, such as support assistants, document question answering, coding agents and batch classification jobs.

Examples

• A support chatbot puts a 4,000 token product manual and tone guide at the top of every request so only the customer message changes.
• A coding agent caches the repository summary and tool definitions across a multi step task, paying full price only for each new instruction.
• A classification job reuses a cached block of 30 labelled examples while feeding in a different item to classify on each call.

How it is measured

• Cache hit rate: share of input tokens served from cache versus computed fresh
• Cost per request before and after caching, split by cached and uncached input tokens
• Time to first token for cache hits compared with cache misses
• Cache invalidation rate caused by prefix changes such as injected variables