Serving & batching

The bottleneck during generation is memory bandwidth, not compute. Reading the weights is the expensive part; multiplying them by one token's activations is not.

So when you batch eight requests, you read the weights once and do eight times the arithmetic — and since the arithmetic was never the constraint, the step takes only slightly longer. Eight times the output for perhaps 1.2 times the time.

That relationship holds until the arithmetic does become the constraint, which happens somewhere in the tens of concurrent requests depending on hardware. Past that point, adding requests raises per-token latency roughly linearly and throughput flattens. The curve has a knee, and finding it on your hardware is the main tuning exercise.

Prefill behaves differently. It is already compute-saturated by a single long prompt, so batching prefill gives much less. This is why serving systems treat the two phases separately, and why some disaggregate them onto different hardware pools entirely.

Static batching collects N requests, runs them together, and returns when the longest finishes. The flaw is visible as soon as you draw it: a request needing 80 output tokens sits in a batch with one needing 800, and its slot is held — producing nothing — for 720 steps.

Continuous batching evaluates the batch at every decode step instead. A finished request leaves immediately and a waiting request joins in its place at the next step. The GPU is never holding an idle slot.

The improvement over static batching is typically two to four times on realistic traffic, and it is larger the more variable your output lengths are. Every modern serving stack does this — vLLM, TGI, TensorRT-LLM — and if you are running your own inference and not using one of them, this is the first thing to fix.

Paired with it is paged attention, which stores the KV cache in fixed-size blocks rather than one contiguous per-request allocation. Without it, each slot must reserve memory for the longest output it might produce, which wastes most of the cache. With it, memory is allocated as tokens are actually generated. The two features together are what made high-concurrency serving practical.

Not compute. Almost always KV cache memory.

GPU memory        80 GB
weights (8B, fp16)     16 GB
overhead                4 GB
available for cache    60 GB
per request at 4k ctx   0.54 GB
max concurrent          ~111

A serving system refusing new requests while GPU utilisation reads 45% is not broken. It is out of cache. This is the single most common confusion when operating an inference server, and the metric to watch is cache occupancy rather than utilisation.

Which means the levers on capacity are mostly memory levers:

Shorter contexts — cache is linear in sequence length, so halving context doubles concurrency.

Cache Quantization — 8-bit keys and values halve the per-request footprint for a small quality cost, and for long-context workloads this is usually the highest-leverage change available.

Weight Quantization — frees memory for cache and speeds decoding, because there is less to read per token.

Grouped-query attention — an architectural property of the model rather than a knob, but it is why a model advertised as long-context-efficient actually is.

Throughput and per-request latency move in opposite directions under load, and no configuration serves both.

Large batches mean excellent throughput, low cost per token, and worse time to first token because new requests queue behind a full batch. Small batches mean the reverse.

The resolution is not a clever setting; it is separate pools. An interactive endpoint runs small batches and aggressive admission control. A batch pipeline runs large batches and accepts queueing. Trying to serve both from one deployment produces a configuration that is wrong for both, and this is a capacity-planning decision rather than a tuning one.

Two controls belong on the interactive path specifically. Cap maximum output tokens, because tail latency here is partly a property of what the model decided to say. And use prompt caching so shared prefixes skip prefill entirely, which removes queueing pressure at its source.

Speculative decoding is worth noting as the one technique that improves latency without hurting throughput much: a small draft model proposes tokens, the large model verifies several at once. Output is identical, so it is a pure win where the draft model's acceptance rate is high.

An 8B model on one A100 80GB, interactive chat, 4k contexts, 300-token answers. Baseline is a naive loop with static batch size 8:

concurrency                          8
throughput                      310 tokens/s
p50 time to first token         1.4 s
GPU utilisation                  31%
cost per 1M output tokens       $2.60

Switch to a continuous-batching server with paged attention:

concurrency                        111   (cache-limited)
throughput                    2,850 tokens/s   (9.2x)
p50 time to first token         0.9 s
GPU utilisation                  78%
cost per 1M output tokens       $0.28   (-89%)

Same model, same hardware, same quality. Then add 8-bit cache quantization:

per-request cache      0.54 GB -> 0.27 GB
concurrency                111 -> 222
throughput             2,850 -> 3,900 tokens/s
p99 latency            rises from 4.1 s to 6.8 s   <- the knee

The last line is the one to notice. Throughput kept rising and the tail got meaningfully worse, because concurrency has passed the compute knee. For a batch pipeline that is a good trade. For an interactive product it is not, and the right setting is somewhere below the maximum the memory allows.