KV cache

In one line

Per-request memory that turns quadratic regeneration into linear growth, and then becomes the thing that runs out first.

Why it exists

Generating token 500 requires attending over tokens 1 to 499. Without a cache, that means recomputing the key and value vectors for all 499 of them — and then doing it again for token 501, and 502. Total work grows with the square of the sequence.

But those vectors do not change. Token 12's key is a function of token 12 and everything before it, both of which are already fixed. The simply keeps them. Each new then computes one new key and value, appends them, and attends over the stored history.

This is straightforward memoisation of a pure function, and it is not optional: without it, long-form generation is computationally infeasible rather than merely slow.

The cache is what makes generation linear instead of quadratic. It is also the memory that decides how many users fit on one GPU.

What is actually stored#

For every layer, every and every position so far, two vectors: the key and the value. Queries are not cached, because a query is only used at the step that produces it and never again.

The size is fully determined by the architecture and the sequence length:

bytes = 2 x layers x kv_heads x head_dim x seq_len x bytes_per_value

The leading 2 is keys and values. Everything else is a fixed property of the model except seq_len, which is yours. That makes cache size linear in context length and per request — two properties that together decide your serving economics.

Grouped-query attention is worth understanding here because it exists purely to shrink this number. Instead of every query head having its own key and value heads, groups of query heads share one set. A model with 32 query heads and 8 key-value heads has a quarter of the cache footprint of one with 32 of each, with very little quality cost. When you read that a model supports long context efficiently, this is usually the mechanism.

Why it becomes the constraint#

Weights are a fixed cost paid once per GPU. Cache is a variable cost paid per concurrent request, and it is the term that scales with your traffic.

On an 80GB card serving an 8B model at 16-bit precision, weights take 16GB and overhead maybe 4GB, leaving 60GB. At 0.5GB of cache per 4,000-token request, that is 120 concurrent requests. The same card at 32,000 tokens of context per request holds 15.

That relationship — traded directly against concurrency — is the central serving tension, and it is why providers price long context the way they do. It also explains something that otherwise looks arbitrary: a serving system that refuses new requests while GPU utilisation looks low. It is not out of compute; it is out of cache.

Two mitigations matter in practice. Paged attention stores the cache in fixed-size blocks rather than one contiguous per-request allocation, which removes the fragmentation and over-reservation that otherwise wastes a large fraction of it — the same idea as virtual memory paging, applied to the same problem. And cache stores keys and values at 8 bits instead of 16, halving the footprint for a small quality cost, on top of any weight quantization.

Prefix caching, across requests#

Everything above is per-request state that dies when the request ends. There is a second, related idea worth separating: reusing cache between requests.

If two requests share a prefix — the same system prompt, the same few-shot examples, the same retrieved document — the KV entries for that prefix are identical. stores them and lets the second request skip for the shared portion entirely.

The economics are strong: providers typically charge a fraction of the normal input price for cached tokens, and time to first token drops sharply. The constraint is that the match must be an exact token prefix. One changed character near the start invalidates everything after it.

That gives you a concrete prompt-layout rule. Stable content first — system prompt, tool definitions, examples, long reference documents. Volatile content last — the user's message, timestamps, request ids. Putting the current time at the top of a prompt is an expensive habit, and it looks harmless.

The math

Cache size for one request:

S = 2 \times L \times H_{kv} \times d_{head} \times n \times b

For an 8B model with $L = 32$ layers, $H_{kv} = 8$ grouped key-value heads, $d_{head} = 128$ , 16-bit values so $b = 2$ , at $n = 4096$ tokens:

S = 2 \times 32 \times 8 \times 128 \times 4096 \times 2 = 5.4 \times 10^{8}

About 537MB per request. Multiply by concurrency to get your real memory requirement, and notice that it is comparable to the weights themselves at quite modest batch sizes.

Worked example

A chat product, 8B model, one 80GB GPU, and the question is how many users it holds.

weights                8e9 x 2 bytes                = 16 GB
runtime overhead                                    =  4 GB
available for cache    80 - 20                      = 60 GB

Three context regimes, same hardware:

4k context    0.54 GB/req    ->  111 concurrent
16k context   2.15 GB/req    ->   28 concurrent
64k context   8.59 GB/req    ->    7 concurrent

Sixteen times the context, sixteen times fewer users. Now apply two changes:

8-bit cache quantization    halves per-request cost
4k context, quantized       0.27 GB/req  ->  222 concurrent

And a prompt-layout change, for a product where every request shares a 1,800-token system prompt and tool schema:

without prefix caching   2,000 prompt tokens billed and prefilled per request
with prefix caching        200 prompt tokens prefilled, 1,800 served from cache
                           TTFT roughly 10x lower, input cost a fraction

Neither change touched the model. Both came from understanding what the cache holds and when it can be shared.

Gotchas

Sizing a deployment on weights alone. A model that "fits in 24GB" fits at batch size one. Under real concurrency the cache is comparable to or larger than the weights, and it is what triggers out-of-memory errors in production after a clean staging run.
Putting volatile content at the front of the prompt. A timestamp, a request id or a user name before the system prompt invalidates the entire cached prefix on every call. Cost and time to first token both jump, and nothing in your metrics points at the cause.
Assuming prefix caching survives anything. It is an exact token-prefix match with a provider-defined time-to-live, usually minutes. Low-traffic endpoints may never hit it at all, and a whitespace change in a template silently ends the streak.
Treating long context as free because the window allows it. The window is an architectural limit; the cache is an economic one. Filling a 128k window can cut your concurrency by an order of magnitude and raise pressure enough to hurt for every other user on the box.

Mental model

It is a per-connection session buffer, exactly like the write buffer a database keeps per open transaction. Individually small, collectively the thing that decides your connection limit, and invisible in every dashboard until you run out of it. The difference from an ordinary cache is that evicting it is not a performance hit — it is a correctness one, because the request cannot continue without it.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

How many users fit on one GPU?
The weights are the fixed cost and the KV cache is the variable one; capacity is a memory calculation you can do on a napkin, and the answer changes by 16x with context length.
after phase 2 · 4 min · #capacity #memory #serving

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Quantization