Back to Mind & Machine

KV Cache Deep Dive

How LLM Providers Cache Their Way to Speed (and Save You Money)

Jiuhe Wang  |  Apr 12, 2026  |  ~12 min read

The Speed Secret Nobody Talks About

Every time you send a prompt to an LLM, there's a hidden hero working behind the scenes: KV Cache. It's the reason your second message in a long conversation is cheaper than the first one, the reason cached context feels "instant," and the reason providers can offer batch processing at lower costs.

But here's the thing — most people don't know how it actually works. They just know it saves tokens. This article fixes that. We'll look at the transformer architecture from an inference angle, trace exactly what happens token by token, and understand why KV cache is the reason your API bill isn't 10x higher.

Core Idea: Instead of re-computing the attention states for all previous tokens every time a new token is generated, LLMs cache these Key and Value matrices. When generating the next token, only the new token's K/V need to be computed. Everything else? Already cached.

Why Do We Even Need KV Cache?

Let's talk about the attention mechanism — the heart of every transformer model. When an LLM processes text, every token needs to "look at" every other token to understand context. This is called self-attention.

O(n2)
Attention Complexity
100B+
Parameters in GPT-4 class models
90%+
Time in matrix multiplication

Here's the painful math: for a sequence of n tokens, computing attention is O(n2) in computation and memory. In autoregressive generation, we process n tokens during prompt filling, then generate token-by-token. Each generated token needs to attend to all n previous tokens.

Without KV Cache

Token 1: compute attention over 1 token

Token 2: recompute attention over 2 tokens

Token 3: recompute attention over 3 tokens

...and so on, each step starting from scratch

Computation at step 10:
10 tokens
With KV Cache

Token 1: compute and cache K/V

Token 2: retrieve cached + compute new

Token 3: retrieve cached + compute new

...cached values reused every step

Computation at step 10:
1 new

Inside the Attention Mechanism

To understand KV cache, we first need to look at what K and V actually are. Every token in a transformer gets transformed into three vectors through learned linear projections:

How Each Token Becomes Q, K, V
Token
"Hello"
Input Token
WQ
Query Projection
Q
Query Vector
Token
"Hello"
WK
Key Projection
K
Key Vector (cached!)
Token
"Hello"
WV
Value Projection
V
Value Vector (cached!)

The Attention Score Computation, Step by Step

The attention formula looks intimidating, but each piece has a clear meaning. Let's break it down:

Attention(Q, K, V)  =  softmax( Q · KT ÷ √dk ) · V
Step 1: Q · KT

Compare every token's Query with every token's Key. Result: an "attention score" for each pair — how much should token A pay attention to token B?

Step 2: ÷ √dk

Scale down by the square root of the key dimension. Prevents the dot products from growing too large, which would make softmax produce extreme probabilities.

Step 3: softmax()

Convert scores into probabilities (all add to 1.0). A token might pay 40% attention to token A, 35% to B, 25% to C, etc.

Step 4: · V

Multiply the attention weights by the Value vectors. This produces a weighted "context" — each token's representation is now infused with meaning from all the tokens it attended to.

Attention Heatmap: "The cat sat on mat"
The
cat
sat
on
mat
The
0.35
0.15
0.20
0.10
0.20
cat
0.15
0.40
0.20
0.08
0.17
sat
0.05
0.18
0.45
0.12
0.20
on
0.08
0.05
0.15
0.52
0.20
mat
0.12
0.08
0.15
0.22
0.43

Hover cells to inspect. Each row shows how much a token (left) attends to each token (top). Diagonal = self-attention.

Let's compute the output for "sat" step by step
Step 1 Q · KT — How much does "sat" relate to each token?
Multiply "sat"'s query vector (Qsat) with every key vector. High score = strong relationship. Here we show the raw dot-product scores before scaling.
Qsat
=
Qsat
0.2
0.1
-0.3
0.4
×
KThe
1.2
×
Kcat
0.9
×
Ksat
2.8
×
Kon
0.4
×
Kmat
0.6
Raw scores: [1.2, 0.9, 2.8, 0.4, 0.6]
scale by √dk  
Step 2 ÷ √dk — Scale down to prevent softmax saturation
If key vectors have large magnitudes, dot products explode. Dividing by √dk (e.g. √64 = 8) keeps values in a reasonable range. This is the "scaled" attention trick.
The
1.2→0.15
cat
0.9→0.11
sat
2.8→0.35
on
0.4→0.05
mat
0.6→0.08
softmax (convert to probabilities)  
Step 3 softmax() — Turn scores into probabilities (sum = 1.0)
Softmax applies exp(x) to each score, then normalizes. High scores get exponentially more weight. exp(0.35) dominates over exp(0.05), so "sat" attends mostly to itself.
The
5%
cat
18%
sat
45%
on
12%
mat
20%
weighted sum · V  
Step 4 · V — Compute the weighted context vector
Multiply each attention weight by its Value vector and sum. "sat"'s new representation = 15%·VThe + 18%·Vcat + 45%·Vsat + 12%·Von + 20%·Vmat
× 0.05
VThe
→
contributes 5% of its meaning
× 0.18
Vcat
→
contributes 18% of its meaning
× 0.45
Vsat
→
dominant contributor (45% self-attention)
× 0.12
Von
→
contributes 12% of its meaning
× 0.20
Vmat
→
contributes 20% of its meaning
Attention Output for "sat": [0.18, 0.24, 0.31, 0.09, 0.18] — A new vector that captures "sat" in context of the full sentence
The Insight: K and V vectors are deterministic — the same token always produces the same K and V vectors, regardless of context. This is what makes caching possible! The Q vector is what changes based on context (what other tokens you're attending to), but K and V are "fixed" per token.

What Exactly Is KV Cache?

KV Cache is exactly what it sounds like: a cache that stores the Key and Value matrices for every token that's already been processed. Instead of recomputing K and V for all previous tokens during each generation step, we simply look them up from the cache.

The Two Phases of LLM Inference

Phase 1: Prefill (Prompt Processing)

The model processes the entire input prompt. For each token, it computes Q, K, and V vectors through all transformer layers. These K and V matrices are stored in the KV cache. This is a parallel computation — all tokens are processed at once.

"What's the"
"weather"
"like?"
K1, V1
K2, V2
K3, V3
Phase 2: Decode (Autoregressive Generation)

The model generates tokens one at a time. For each new token, it computes Q (unique to this position) and reuses cached K/V from all previous positions. This is sequential — one token at a time.

K1, V1
K2, V2
K3, V3
Q4
Output Token

The Speedup Numbers

Let's put numbers on this. For a 1000-token conversation where we generate 100 tokens:

Scenario K/V Computations vs Baseline
No KV Cache (baseline) 1000 + (100 × 1000) = 101,000 1x
With KV Cache 1000 + (100 × 1) = 1,100 ~92x faster!
Prompt Cached (reuse) 0 + (100 × 1) = 100 ~1000x faster!
The Key Insight: During decoding, we go from O(n) K/V computations per token to O(1) — we just look up the cache. This is the entire speedup from KV cache. The attention matrix computation still grows with sequence length, but the K/V matrix multiplications become free.

Step-by-Step Decoding with KV Cache

A tiny model processes a 5-token prompt and generates 4 tokens. We'll walk through every step — what gets computed, what gets read from cache, and how the cache grows.

Example: "The cat sat on mat"
T1: "The"
T2: "cat"
T3: "sat"
T4: "on"
T5: "mat"
GEN: "and"
GEN: "purred"
GEN: "happily"
GEN: "."

Prefill (5 tokens: "The cat sat on mat") → Decode (4 tokens: "and purred happily.")

Phase 1: Prefill — Processing the Prompt

During prefill, the model processes all 5 input tokens in parallel. For each token at each layer, it computes Q, K, and V, then stores K and V in the cache.

Token Compute Q Compute K, V Store in Cache
T1 "The" Q1 computed K1, V1 computed Cache += [K1, V1]
T2 "cat" Q2 computed K2, V2 computed Cache += [K2, V2]
T3 "sat" Q3 computed K3, V3 computed Cache += [K3, V3]
T4 "on" Q4 computed K4, V4 computed Cache += [K4, V4]
T5 "mat" Q5 computed K5, V5 computed Cache += [K5, V5]

After prefill: the cache holds 5 KV pairs — one for each token. Now the decode phase begins.

After prefill, the cache contains: [K1, V1, K2, V2, K3, V3, K4, V4, K5, V5] — for all 5 tokens, for all layers.

Phase 2: Decode — Generating Token by Token

Now the model generates tokens one at a time. At each step, it reuses cached K/V and computes only the new token's Q. Here's what that looks like over 4 decode steps:

Decode Step 1: Generate "and"

The model computes attention between the new token and all 5 cached tokens.

Retrieve from cache: K1..5 V1..5 (5 KV pairs retrieved)
Compute fresh: Q6 (1 Q computed)
Save new: K6 V6 → added to cache

Cache now contains: 6 KV pairs (5 from prefill + 1 new)

Decode Step 2: Generate "purred"

Cache now holds 6 KV pairs. Model retrieves all of them.

Retrieve from cache: K1..6 V1..6 (6 KV pairs retrieved)
Compute fresh: Q7 (1 Q computed)
Save new: K7 V7 → added to cache

Cache now contains: 7 KV pairs. Notice: K1 through K6 were not recomputed — just read from memory.

Decode Step 3: Generate "happily"

Same pattern. Cache grows by one more pair.

Retrieve from cache: K1..7 V1..7 (7 KV pairs retrieved)
Compute fresh: Q8 (1 Q computed)
Save new: K8 V8 → added to cache

Cache now contains: 8 KV pairs

Decode Step 4: Generate "." — done!

Final step. The model decides to stop.

Retrieve from cache: K1..8 V1..8 (8 KV pairs retrieved)
Compute fresh: Q9 (1 Q computed)
Save new: K9 V9 → added to cache. Done!

Final cache: 9 KV pairs. The full sentence "The cat sat on mat and purred happily." is complete.

Computation Summary
Phase Q Computed K/V Computed Cache Reads Cache Writes
Prefill (5 tokens) 5 5 0 5
Decode (4 tokens) 4 4 5+6+7+8 = 26 4
Total 9 9 26 9

Without KV cache, each decode step would recompute all K/V from scratch. That's 5 + 6 + 7 + 8 = 26 total K/V computations. With cache: only 9. That's 3x fewer K/V computations. As the generated sequence gets longer, the savings grow — at step 10 you'd be retrieving 14 KV pairs but computing only 1.

Multi-Layer Reality

Real LLMs have dozens of transformer layers. Each layer has its own K and V projections — meaning each layer maintains its own KV cache. For a 32-layer model, you actually have 64 KV caches running in parallel!

Layered KV Cache Architecture
Layer 32
K32, V32 caches
Layer 16
K16, V16 caches
Layer 1
K1, V1 caches

Each layer stores K/V for every token. For a 32-layer model with 8 tokens in context: 32 layers × 2 (K & V) × 8 tokens = 512 cache entries.

Real-World Example: A Multi-Turn Conversation

Let's trace a practical scenario: a user has a 5-message conversation with a coding assistant. We'll see how the KV cache evolves across messages and what it means for your wallet.

User
"Write a Python function to calculate fibonacci numbers"
Tokens: 12 input → model generates ~80 tokens response
Cache state: Prefill: compute K/V for 12 tokens → cache has 12 KV pairs
Decode: compute Q for 80 tokens, each retrieving 12..91 KV pairs from cache
User
"Can you add memoization?"
Tokens: 8 input (with conversation history attached) → model generates ~40 tokens
Cache state: Prefill: compute K/V for 8 tokens, but 12 tokens already in cache → only 8 new KV pairs added
Decode: retrieve up to 100 KV pairs (92 cached + 8 new) for each of the 40 decode steps
Savings: 12 tokens from Message 1 are cached! No recomputation needed.
User
"Now add a decorator to time the function"
Tokens: 10 input → model generates ~60 tokens
Cache state: Prefill: 10 new tokens → cache grows from 100 to 110 KV pairs
Decode: each step retrieves up to 110 cached KV pairs
Savings: All 100 KV pairs from previous conversation are reused!
User
"Test it with fib(30)"
Tokens: 5 input → model generates ~20 tokens
Cache state: Cache already has 110 KV pairs. Only 5 new ones needed.
Total KV pairs after this message: 115
User
"Print the result"
Tokens: 4 input → model generates ~15 tokens
Cache state: 115 cached KV pairs. Only 4 new ones added. Total: 119
Each decode step in this message: retrieves 119 KV pairs from cache!
Token & Cost Breakdown
Message Input Tokens Cached from History New KV Computed Cost (cached vs new)
Msg 1 (fibonacci) 12 0 12 + 80 decode = 92 Full price
Msg 2 (memoize) 8 92 8 + 40 decode = 48 ~50% cheaper
Msg 3 (decorator) 10 100 10 + 60 decode = 70 ~55% cheaper
Msg 4 (fib 30) 5 110 5 + 20 decode = 25 ~60% cheaper
Msg 5 (print) 4 115 4 + 15 decode = 19 ~65% cheaper

By message 5, most of the work is already done. Those 115 cached KV pairs from earlier messages mean message 5's decode steps barely touch new computation. Compare that to sending message 5 as a standalone request with no history — you'd pay full price for everything.

Prefix Caching: The Next Level

Prefix caching extends session-level KV cache across multiple requests. Instead of caching within a single conversation, it caches based on the content of the prompt prefix. If two requests share the same system prompt or instruction prefix, they share the KV cache — no recomputation needed.

The Hash-Based Approach

Providers compute a cache key (usually a hash) from the prefix content. When a new request arrives, the system checks if any cached prefix matches. Matching bytes can reuse cached KV states — no recomputation needed.

Prefix Cache Hit Scenario
Request A (First time):
System: You are a helpful assistant...
User: Explain quantum computing...
COMPUTE + CACHE
Request B (Same system prompt):
System: You are a helpful assistant...
CACHED!
User: What is entanglement?
PREFIX CACHE HIT!

What Gets Cached?

Different providers cache at different granularities:

Cache Type Cached Content Savings
Session Cache Same conversation thread 50-90% on continuation
System Prompt Cache Fixed system instructions Per-request savings
Semantic Prefix Common instruction patterns Cross-request savings
Tool Definition Cache Function schemas, RAG docs Significant for tool-use

The Memory Math: Why KV Cache Is Expensive

KV cache is brilliant for speed, but it's a memory monster. Every cached K/V vector consumes GPU VRAM. Let's calculate the real cost.

KV Cache Memory Formula — Plain English

Here's what each piece of the formula means in plain terms:

2 (K + V) × 80 layers × 8192 tokens × 8192 values per token × 2 bytes
2 (K + V)
We store both Key and Value matrices for each token. Two matrices per token.
80 layers
Each transformer layer has its own K and V projections. So we have 80 independent cache stores.
8192 tokens
Maximum sequence length. Each token needs its own K and V entry in every layer.
8192 values
Hidden dimension size. Each K/V vector has 8192 floating-point numbers.
2 bytes (FP16)
Half-precision float. 2 bytes per number instead of 4 bytes for full precision.

The punchline: 2 × 80 × 8192 × 8192 × 2 bytes = ~21 GB for a single full-context sequence on Llama 3 70B.

Real-World Example: Llama 3 70B

80
Transformer Layers
8K
Context Window
8K
Hidden Dimension
32
Attention Heads
128
Head Dim
Step-by-Step Calculation
Per token per layer (FP16):
= 2 (K + V) × 8192 (hidden dim) × 2 (bytes)
= 32,768 bytes = ~32 KB per token per layer

Full context (80 layers × 8192 tokens):
= 32 KB × 80 layers × 8192 tokens
= ~21 GB per sequence
What this means for you: A single A100 (80GB VRAM) can hold roughly 3-4 KV caches for Llama 3 70B at full 8K context. That's why providers limit how many long-context requests you can run in parallel. Memory, not compute, is what constrains batch size.

Memory Optimization Techniques

  • FP16 vs FP8: Halving precision halves cache size — almost no quality loss
  • KV Cache Quantization: Store cache in INT8 or even INT4 — significant memory savings
  • PagedAttention: vLLM's innovation — manage KV cache like virtual memory pages, eliminating fragmentation waste
  • Streaming cache: Evict older tokens from cache when context window is exhausted (sliding window)
  • Cross-request sharing: Prefix caching lets multiple requests share the same physical cache entries

    • How Different Providers Implement KV Cache

    Every major LLM provider has their own KV cache implementation. Let's break down what each one offers:

    OpenAI GA
    Automatic prompt caching via cache_controls
    Reduced pricing on cached tokens (10x cheaper)
    Automatic prefix detection
    Works with o1, GPT-4o, GPT-4.5
    No explicit cache control in older APIs
    Anthropic GA
    Automatic KV caching (session-level)
    Cache-beta pricing (50% discount on cached)
    Extended thinking mode with cache
    Claude 3.5+ and Claude 4
    No cross-session prefix caching
    Google Gemini NEW
    Context caching with explicit cost
    Charged per 1M tokens + retrieval
    Best for repeated RAG contexts
    Supports up to 2M context
    Manual cache key management needed
    Groq / Cerebras Fast
    LPU architecture — KV cache on-chip
    Massive throughput (1000+ tokens/sec)
    Minimal cache miss penalty
    Free automatic caching
    Smaller model selection
    Together AI / Fireworks GA
    Prefix caching API available
    Custom model deployment
    Streaming prefix detection
    Pay-per-token with cache discount
    Open model support (Llama, Mistral)
    Grok / xAI Beta
    Basic session caching
    Integrated with X platform
    Lower pricing model
    Limited caching features
    Less transparent documentation

    KV Cache and Cost Optimization

    This is where KV cache directly impacts your wallet. Understanding cache mechanics lets you structure prompts for maximum reuse.

    Cache Hit = Massive Savings

    Cache Hit

    Same prefix is reused

    System prompt: ~$0.003/1K tokens
    Cached tokens: ~$0.0003/1K tokens (90% off!)
    Best for: Chatbots, agents with system prompts, tool-using with fixed schemas
    Cache Miss

    No matching prefix found

    All tokens: ~$0.03/1K tokens (full price)
    Watch out: Unique prompts each time = no savings. RAG with different docs = miss every time.

    The Cache Temperature Model

    Think of KV cache as having a "temperature" — how likely it is to hit:

    Hot Cache
    Same conversation, recent turns. 80-95% hit rate
    Warm Cache
    Same system prompt, different users. 50-80% hit rate
    Cold Cache
    Unique prompts, first-time requests. 0-20% hit rate

    Real Example: Agentic AI System

    Consider an AI coding assistant with this prompt structure:

    # System Prompt (static, cached) "You are a senior software engineer. Follow best practices..." ~500 tokens # Tool Definitions (semi-static, cached) "Available tools: read_file, write_file, run_command..." ~300 tokens # RAG Context (dynamic, unique per request) "Codebase context from embeddings..." ~2000 tokens # User Message (unique) "Fix the authentication bug in login.ts" ~10 tokens # Cache Analysis: # System + Tools: 800 tokens cached every time = 90% discount # RAG + User: 2010 tokens at full price # Effective savings: ~40% on total token cost!

    Practical Tips & Tricks

    Maximizing Your Cache Hit Rate

    Tip 1
    Structure Your Prompts with Cache in Mind

    Put static content first — system instructions, tool definitions, guidelines. Put dynamic content last. The longer the matching prefix, the more cache hits.

    Tip 2
    Use Cache Control Markers (OpenAI)

    With OpenAI's explicit caching, wrap content in cache control markers. The provider will tell you exactly what's cached.

    # Mark sections as cacheable "<cache>Static instructions here</cache>" "<cache>Tool definitions here</cache>" "<cache>Guidelines here</cache>" # These sections will be cached and reused
    Tip 3
    Batch Similar Requests

    If you have multiple requests with similar prefixes (e.g., analyzing different files with the same instructions), batch them together or send them sequentially to leverage session caching.

    Tip 4
    Keep Conversations Under Cache TTL

    Most providers expire session caches after 5-60 minutes of inactivity. If you're building a long-running agent, keep sessions alive or explicitly manage cache state.

    Tip 5
    Measure Your Actual Cache Hit Rate

    Don't guess. Most providers return cache hit metadata in their API responses. Track this in your observability stack. A 70% cache hit rate on an agent making 10K requests/day = thousands of dollars saved.

    The Complete Picture

    Here's the KV cache lifecycle:

    1. Prefill: Input tokens processed in parallel. K/V computed and cached for every token at every layer.

    2. Decode: Each new token computes only Q — K and V are fetched from cache. Massive speedup.

    3. Prefix Cache: If the same prefix appears in multiple requests, the K/V are shared across requests. Massive cost savings.

    4. Memory: All that cached data lives in GPU VRAM. Cache size grows with sequence length and model size. Memory is the bottleneck.

    5. Optimization: Providers use quantization, paging, and smart eviction to maximize cache efficiency. You just pay less.

    Key Takeaways

    • KV cache is why LLM inference doesn't take forever — without it, generating a 500-token response would require recomputing attention from scratch at every single step
    • Session-level caching is automatic on most providers — keep a conversation going and the per-message cost drops fast
    • Prefix caching is where providers compete now — OpenAI, Anthropic, and Google all want to be the one caching your system prompts
    • Memory is the actual bottleneck — KV cache size scales with sequence length and model size. Every extra token in context is more GPU RAM eaten
    • Prompt structure drives your cache hit rate — static content first, dynamic last, and your bill reflects it
    • Groq for speed, OpenAI for features, Anthropic for quality, Gemini for long context — each has its own cache implementation

    Now that you understand KV cache, you'll never look at your API bill the same way again.