ExampleBest after Phase 24 min

How many users fit on one GPU?

In one line

The weights are the fixed cost and the KV cache is the variable one; capacity is a memory calculation you can do on a napkin, and the answer changes by 16x with context length.

The situation

An internal chatbot is moving from a provider API to a self-hosted 8B model, and someone has to answer "how many concurrent users does one 80GB GPU support?" The first answer offered is "the model is 16GB, so plenty". The first load test disagrees: at forty concurrent users the server starts refusing requests with GPU utilisation at 45%.

The card is not out of compute. It is out of a kind of memory nobody budgeted for.

The weights do not move. Every knob that changes how many users fit on the card is a knob on the cache.

Two kinds of memory#

The weights — all eight billion at 16-bit — take 16GB and are loaded once. That is the fixed cost, and it is what "the model is 16GB" refers to.

Each in-flight request additionally holds a : the attention keys and values for every token in its context so far, so the model does not recompute history at every step. That cost is per request, grows linearly with the request's usage, and is freed when the request ends. It is the variable cost, and at realistic concurrency it is larger than the weights.

For this model, one 4,000-token request holds roughly 540MB of cache. That is the number the plan was missing.

The napkin#

GPU                                 80 GB
weights (8B × 2 bytes)              16 GB
runtime overhead, workspace          4 GB
left for cache                      60 GB
per request at 4k context          0.54 GB
concurrent requests                 ~111

About a hundred and ten users at 4k context. The load test hit a wall at forty because the test used 12k-token conversations — three times the cache per request, a third of the concurrency.

Now the levers are visible, and none of them is "a bigger model would be worse". Halve the context per conversation by summarising old turns, and concurrency doubles. Quantize the cache to 8-bit — a separate setting from weight — and it doubles again. Quantize the weights to 8-bit and 8GB more is freed for cache. Each of these is a configuration or prompt change, not a hardware purchase.

The math

Cache per request, from the architecture:

S = 2 \times L \times H_{kv} \times d_{head} \times n \times b

Layers $L = 32$ , key-value heads $H_{kv} = 8$ , head dimension $d_{head} = 128$ , context $n = 4096$ tokens, $b = 2$ bytes at 16-bit:

S = 2 \times 32 \times 8 \times 128 \times 4096 \times 2 \approx 5.4 \times 10^{8} \text{ bytes}

About 537MB. Divide the memory left after weights by this and you have your concurrency. Change $n$ or $b$ and you have your levers.

Questions to ask

What context length does a real conversation reach, not the limit but the median and p95?
Am I monitoring cache occupancy, or only GPU utilisation?
Which is cheaper for me: shorter contexts, a quantized cache, or another card?
Does the model use grouped-query attention? (It changes $H_{kv}$ , and the answer, by 4x.)

Mindset

Capacity is a memory problem before it is a compute problem. Budget the per-request cache first, watch it in production, and remember that every context-length decision in the product is a concurrency decision on the GPU.

Where it connects

Got the shape of it?

Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.

Next: Summarising a 400-page PDF