Concepts · ai
Prompt Caching
Cross-provider cache_control markers, retention levels, session affinity
- Enabling or disabling prompt caching
- Adding caching support to a new provider adapter
- Debugging why cache hits aren't happening
This page gives the prompt caching model across providers. Prompt caching lets a provider reuse the KV cache of a previously processed prefix. The provider does not recompute it from scratch. For repeated system prompts, long tool definitions, or multi-turn conversations, cache hits reduce latency and cost by an order of magnitude. pi-go enables caching automatically on supported providers.
Design: default on, terminal breakpoint
Caching is on by default. Callers who never set an option still get cache hits. No opt-in is required. CacheRetention controls the behavior and has four values:
CacheRetentionDefault(zero value) — resolves toShort.CacheRetentionShort— provider’s default ephemeral TTL (Anthropic: 5 minutes).CacheRetentionLong— longer ephemeral TTL where supported (Anthropic: 1 hour onapi.anthropic.com).CacheRetentionNone— disable markers entirely for this request.
A single helper, ai.ResolveCacheRetention, centralizes the rule that the default is Short. Provider adapters use it to keep behavior aligned.
Design: terminal-breakpoint placement
pi-go places at most two cache breakpoints per request:
- One on the system prompt.
- One on the final content block of the last message in the conversation.
The marker means that everything before this point is cacheable. On the next turn, the previous terminal block is now inside the prefix. It still matches the cached bytes. A single static placement rule delivers turn-over-turn hits. The agent does not need to track where it put the marker last time.
Why not also mark tools. Tool schemas are part of the prefix that gets cached automatically once any downstream marker exists. Anthropic caps cache_control at 4 breakpoints per request. Marking tools spends a slot for no added benefit.
Why not split the system prompt. Claude Code subdivides system prompts into static and dynamic blocks. This means that a volatile suffix does not invalidate a stable prefix. That first-party optimization needs callers to annotate which parts of the system prompt change per turn. pi-go treats the system prompt as a single block. It leaves the split strategy to the caller. Callers can join multiple system chunks themselves when they want finer control.
Per-provider behavior
| Provider | Mechanism | Controlled by |
|---|---|---|
| Anthropic Messages | Native cache_control: {type: "ephemeral", ttl?} blocks | CacheRetention (injection) + base URL gate (TTL) |
| OpenAI Chat | Automatic prefix match + prompt_cache_key affinity | SessionID (affinity only; caching is automatic) |
| OpenAI Responses | Same as OpenAI Chat | SessionID |
| Google (Gemini) | Fully implicit, server-managed | Not configurable; hits reported via CacheRead |
Anthropic is the only first-party provider in pi-go where the SDK emits markers. OpenAI’s caching is automatic once requests share a prefix. pi-go forwards StreamOptions.SessionID as prompt_cache_key to strengthen cross-request affinity. It never emits block-level markers for OpenAI. Google manages caching server-side and does not expose a client control.
The 1h TTL for Anthropic is attached only when the client talks directly to api.anthropic.com. Proxies and compatible endpoints receive the marker without a TTL field. This keeps the request serializable against third-party servers that do not understand the extension.
Usage tracking
Cache hits and writes are reported in ai.Usage as separate CacheRead and CacheWrite token counts. CalculateCost multiplies them by Model.Cost.CacheRead and Model.Cost.CacheWrite. Cost breakdowns therefore distinguish fresh input, cache write, and cache read. Anthropic reports both read and write tokens. OpenAI reports only cache reads via prompt_tokens_details.cached_tokens. Google reports cache reads via cachedContentTokenCount.
Adding markers in a new provider adapter
The Anthropic adapter in pkg/ai/provider/anthropic is the canonical example. When you wire a new provider:
- Call
ai.ResolveCacheRetention(opts.CacheRetention)at the top of the request builder. - If the result is
CacheRetentionNone, emit no markers and noprompt_cache_key. - Compute provider-specific TTL only when the client’s configured base URL matches the official endpoint.
- Give proxies the marker without TTL, or no TTL at all, depending on the provider.
- Build one marker value and attach it to the system prompt block.
- Walk the converted messages, find the last content block of the last message, and attach the same marker. For a union type, branch on whichever content type is populated, such as text, tool-result, or image.
- Extract
CacheReadandCacheWritefrom the response usage intoai.Usageso cost calculation works.
Session affinity
StreamOptions.SessionID, set via ai.WithSessionID, provides a stable identifier for cache affinity. Today it matters only for OpenAI. Both the Chat Completions and Responses adapters forward it as prompt_cache_key. Other providers ignore it. Auto-generation is opt-in. Callers create a UUID and pass it themselves. When CacheRetentionNone is set, SessionID is suppressed too. A client can fully turn off cache-related wire-level fields.
Claude Code’s approach of pure prefix matching works because OpenAI’s cache key is only an affinity hint. It uses no session ID. Byte-identical requests still hit the cache without it. The session ID is a strengthening signal, not a requirement.
Disabling
Pass ai.WithCacheRetention(ai.CacheRetentionNone) on a specific call. For an agent, thread it through with agent.WithStreamOpts(ai.WithCacheRetention(ai.CacheRetentionNone)). There is no environment variable. All control is through the options API.