Prompt Caching

Blog
Documentation
Careers
Support
Contact Sales

AI and Machine Learning Solutions
Cloud and Networking Solutions
Database and Storage Solutions
Security and Compliance Solutions

Tutorials and Guides
FAQs and Troubleshooting
Community Forum

Table of Contents for Prompt Caching Tutorial

Authors: Jeff Fan and Anish Singh Walia discuss prompt caching mechanics and a real-world measurement on DigitalOcean Serverless

Prompt caching mechanics apply across providers, with a case study from ProjectDiscovery showing a failure mode and a fix, resulting in a significant increase in cache hit rate and reduction in inference costs

A single structural change in the prompt, moving a dynamic identifier from the middle to the end, increased the hit rate from 7% to 74% and reduced the monthly inference bill by 59%

The pattern is common: caching is configured but one dynamic field in the wrong position breaks the entire prefix, and this article covers how prompt caching works and how to achieve high hit rates in production

Prompt caching is a high-leverage cost optimization for agentic and multi-turn workloads, with a production case study showing a 7% to 74% hit rate increase and 59% inference cost reduction
Three different caches solve different problems: KV cache, prefix cache, and semantic cache
The cache is keyed on exact, byte-identical prefix content, with one dynamic field invalidating the entire prefix behind it
Anthropic-style explicit caching has a write premium that pays back after about 1 cache hit, with every hit after that being pure savings

Understanding KV, Prefix, and Semantic Caching

Caching in the LLM inference context means three different things, and confusing them leads to misplaced optimization effort

1. KV cache is automatic and scoped to a single request, storing key-value attention states for each input token to speed up generation

2. Prefix cache, or prompt cache, is cross-request and controlled by the user, extending the KV cache concept across multiple requests and reusing computed KV states from the first request

3. Semantic cache is an application-layer solution, storing complete input-output pairs and retrieving them when a new query is semantically similar to a previous one

You can learn more about Prompt Caching and its applications

Three different caches at three different layers: KV, prefix, and semantic, each solving a distinct problem

Cache Reads and Writes: Understanding the Economics

The economics of prefix caching make it a dominant cost optimization, with cache reads costing 10% of input and the write premium paying back after about 1 cache hit

On Anthropic, a cache write costs 1.25 times the base input rate, and a cache read costs 0.10 times the base input rate, resulting in significant savings with multiple cache hits

OpenAI's cached-input discount varies by model generation, and checking the rate for the specific model is essential to understand the economics of prefix caching

The write cost is the variable to understand, and caching isn't free, but the math works if subsequent requests hit the cache often enough

When prefix caching helps and when it doesn't, including the impact of prefix length and traffic interval on caching effectiveness

Prefix caching helps with a large stable prefix reused across many requests within the TTL window, resulting in significant cost savings
Prefix caching doesn't help with prefixes below the provider's minimum cacheable length or with low-reuse prefixes where the write premium never amortizes

The Impact of Dynamic Fields on Prefix Caching

A dynamic field in the prefix drops the hit rate to single digits, which is a common mistake made by teams implementing caching for the first time

The prefix cache requires bit-for-bit identical content from the beginning of the prompt, and any change resets the cache boundary at that point

Moving a dynamic field from the middle of the prompt to the end can significantly increase the cache hit rate, as seen in the ProjectDiscovery case study

Same content, one field moved, and the impact on caching, with the stable layout resulting in a higher cache hit rate

The ProjectDiscovery team's hit rate increased from 7% to 74% by moving a dynamic identifier to the end of the prompt, resulting in significant cost savings

Four Steps to Achieve 60-80% Hit Rate with Prefix Caching

Step 1: Identify your stable prefix and map your prompt structure to categorize every component

Your cacheable prefix is the longest leading sequence that stays identical across the majority of requests

Step 2: Move all dynamic content to the end, restructuring the prompt to put the stable prefix first and the dynamic content after the marked cache boundary

[ System instructions — fully static ] [ Tool definitions — fully static ] [ Few-shot examples — fully static ] [ Retrieved context — stable template, variable content ] [ User query — dynamic ] [ Session metadata — dynamic, at the very end ]

Concretely, in the Anthropic-style cache_control API, the trap is a value that changes every request, and the fix is to move it out of the system into the user turn

// The session id is DIFFERENT on every request — watch where it sits. // BROKEN — id inside the cached system block: { "system" : [ { "type" : "text" , "text" : "Session 4f9a-2c1e. You are a support agent. [ …600 lines of stable policy… ]" , "cache_control" : { "type" : "ephemeral" } } ] , "messages" : [ { "role" : "user" , "content" : "How do I reset my password?" } ] } // The next request carries "Session 7b3d-9f02…", so the system text is never // byte-identical twice — the whole prefix is recomputed. Measured: 0% hit. // FIXED — id moved to the user turn; system stays byte-identical: { "system" : [ { "type" : "text" , "text" : "You are a support agent. [ …600 lines of stable policy… ]" , "cache_control" : { "type" : "ephemeral" } } ] , "messages" : [ { "role" : "user" , "content" : "Session 4f9a-2c1e. How do I reset my password?" } ] } // The changing id now rides after the breakpoint, so the 600-line prefix // caches on every later request. Measured: ~99% hit.

Step 3: Monitor cache hit rate as a first-class metric, logging it and alerting on it, as it's a leading indicator of prompt-structure regressions and billing surprises

Step 4: Match cache TTL to your traffic interval, choosing the TTL based on your expected request interval at p50, not the burst peak

Cache-Aware Routing and Its Impact on Latency

The cost savings from prompt caching are well-documented, but the latency impact is less discussed and can be equally significant

Google's Vertex AI team documented that intelligent routing doubled their prefix cache hit rate and cut TTFT by 35% on context-heavy workloads and P95 tail latency by 52% on bursty workloads

The mechanism is that cache hits eliminate the prefill computation for the stable portion of the prompt, resulting in a meaningful latency win

First-Hand Experience with DigitalOcean Serverless

To put numbers to the benefits of prefix caching, we benchmarked DigitalOcean Serverless Inference directly, with two key findings

First, caching on DigitalOcean serverless is explicit, not automatic, and requires marking the stable prefix with an ephemeral cache_control breakpoint

Second, with the prefix correctly marked, moving the per-request dynamic field out of the cached prefix resulted in a dramatic increase in cache hit rate and reduction in input cost

First-hand experience on DigitalOcean serverless showed that marking the prefix and moving the dynamic field to the tail took the hit rate from 0% to 99.3% and cut input cost roughly 90%

One honest caveat from the data is that the reproducible win is cost, not throughput, with TTFT and throughput fluctuating with queue conditions while the cost reduction held steady

Semantic Caching for Repeat Queries

For workloads with high query repetition, semantic caching complements prefix caching by eliminating full inference for common queries

The economics are compelling, with a cache hit having near-zero marginal cost, and semantic caching can eliminate nearly half of all inference calls for a support bot

When to use semantic caching and what to manage, including staleness, threshold tuning, and security

Staleness: cached responses need a TTL or invalidation mechanism
Threshold tuning: too high a similarity threshold yields few hits; too low returns wrong responses for different queries
Security: in multi-tenant deployments, namespace the cache by tenant

Pitfalls Checklist Before You Ship

No dynamic fields are embedded inside the stable prefix
Retrieved documents follow the stable prompt template but appear after the fixed prefix
Cache hit rate is logged per deployment and monitored using alerting

Done right, all three layers together serve 60-80% of inference cost and latency from cache, with only prefix caching depending on prompt structure

Common Questions on Prompt Caching

1. What is prompt caching, and how is it different from a KV cache?

A KV cache is automatic and scoped to a single request, while prompt caching is cross-request and controlled by the user

2. Why is my cache hit rate low even though I've enabled caching correctly?

The most common cause is a dynamic field sitting before the end of the cacheable prefix, and moving it to the end can resolve the issue

3. Does prompt caching change the model's output?

No, caching only affects how the input prefix is processed internally, without changing the response generated by the model

4. What's the minimum prompt length that can actually be cached?

It depends on the provider and the specific model generation, with no single universal number applying across all models and generations

5. How many cache hits does it take before caching actually saves money?

On Anthropic's standard 5-minute-tier rates, the break-even point is about 1 cache hit, after which every additional hit is pure savings

6. Is OpenAI's cached-input discount the same as Anthropic's?

Not uniformly, with OpenAI's cached-read discount varying by model generation and Anthropic's read discount being a consistent 90% off

7. Can prompt caching and semantic caching be used together?

Yes, they solve different problems and stack rather than compete, with prefix caching cutting the cost of reprocessing a shared prompt structure and semantic caching skipping inference entirely for semantically similar queries

8. Does moving a dynamic field to the end of the prompt always fix a low hit rate?

It's the single highest-leverage fix and resolves the majority of cases, but it won't help if the prefix is below the provider's minimum cacheable length or if the traffic is too sparse to produce repeat hits within the TTL window

Prompt caching is a powerful tool for reducing LLM costs and latency, and it can be implemented with minimal effort but significant impact

By following the steps outlined in this article, you can implement prompt caching and start seeing cost savings and latency improvements in your LLM workloads

You can refer to the following tutorials from this Inference in Production series to get started with prompt caching and other optimization techniques

Implement prompt caching in your LLM workloads
Monitor your cache hit rate and adjust your TTL as needed
Experiment with different prompt structures to see what works best for your workload

How Does Prompt Caching Work and When Does It Actually Cut LLM Costs?
How We Cut LLM Costs by 59% With Prompt Caching
How GKE Inference Gateway improved latency for Vertex AI

Learn more about our products and solutions for AI and machine learning

About the author(s)

The authors are experts in AI, machine learning, and cloud computing, with experience in DevOps, Kubernetes, and GenAI

Anish is a Sr Technical Content Strategist and Team Lead at DigitalOcean, with 7+ years of experience in DevOps, SRE, and technical writing

This textbox defaults to using Markdown to format your answer

You can type !ref in this text area to quickly search our full set of tutorials, documentation, and marketplace offerings and insert the link

Featured Tutorials and Guides

All tutorials
All topic tags

Please complete your information to get started with our products and solutions

Table of contents
TL;DR
KV, prefix, and semantic caching solve three different problems

Ubuntu
Linux Basics
JavaScript
Python
MySQL
Docker
Kubernetes

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation

DigitalOcean Documentation and Resources

Full documentation for every DigitalOcean product and service

Resources for Startups and AI-Native Businesses

The Wave has everything you need to know about building a business, from raising funding to marketing your product

The Developer Cloud for AI and Machine Learning

Scale up as you grow, whether you're running one virtual machine or ten thousand

Start Building Today with Our Products and Solutions

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications

About
Leadership
Blog
Careers
Customers
Partners

Knowledge Bases
GPU Droplets
Bare Metal GPUs
Inference Engine

Community Tutorials
Community Q&A
CSS-Tricks
Currents Research

AI Training GPU
GPU Inference
VPS Hosting
Website Hosting

Support
Sales
Report Abuse
System Status

About
Leadership
Blog
Careers
Customers

Knowledge Bases
GPU Droplets
Bare Metal GPUs
Inference Engine

Community Tutorials
Community Q&A
CSS-Tricks
Currents Research

AI Training GPU
GPU Inference
VPS Hosting
Website Hosting

Support
Sales
Report Abuse
System Status