Prefill & decode

The whole prompt is available at once, so every position can be processed in parallel. One pass through the model computes the representations for all 2,000 of your prompt tokens simultaneously, filling the KV cache as it goes.

Because it is a large matrix multiplication with plenty of parallel work, prefill saturates the GPU's arithmetic units. It is genuinely FLOPs-bound, and the standard estimate applies: roughly two operations per parameter per token.

Prefill is what sets time to first token. Doubling the prompt roughly doubles the wait before anything appears, which is the latency a user in a streaming interface actually perceives as "slow to start".

The important property of prefill is that it is cacheable. The computation for a given prefix is deterministic, so if the next request begins with the same 2,000 tokens of system prompt and examples, its KV entries can be reused wholesale. Prompt caching exploits exactly this, which is why stable content belongs at the start of a prompt and volatile content at the end — the same ordering instinct as a Dockerfile.

Once prefill has finished, generation is Autoregressive: produce a token, append it, produce the next. Each step is a full pass through every layer to produce a single token.

Here is the part that surprises people. That pass reads all 16GB of an 8B model's weights from memory in order to compute one token's worth of output. The arithmetic is trivial by GPU standards; the memory traffic is not. Decode is bound by memory bandwidth, and a GPU running a decode step at batch size one is mostly idle, waiting on memory.

This single fact explains most of serving:

Output length dominates latency. A 2,000-token prompt and a 50-token answer is fast. A 200-token prompt and a 2,000-token answer is slow, on the same model.

Inter-token latency is roughly constant per model and load, so total time is predictable: time to first token, plus output tokens times the per-token gap.

Batching is close to free on the decode side. Since the bottleneck is reading weights, and one read can serve every request in the batch, doubling the batch barely changes per-token latency while doubling Throughput. This is the entire basis of continuous batching and the reason serving economics work at all.

Every decode step ends the same way: the model emits Logits — one raw score per vocabulary entry — and something has to pick one token from them. That choice is the only place randomness enters the whole pipeline, and it is entirely under your control.

Softmax turns the scores into a probability distribution. Temperature is a divisor applied before that step: below one it sharpens the distribution toward the likeliest token, above one it flattens it. Top-p then trims the tail, sampling only from the smallest set of tokens whose probabilities add up to p. Greedy decoding is the degenerate case — always take the top token — and it is what temperature zero effectively means.

Two consequences matter in practice. For extraction, classification and anything you will compare in an Eval, you want greedy: the same input then produces the same output, run to run, which is what makes a regression measurable. For prose, greedy produces flat, repetitive text, and a moderate temperature with top-p is the usual setting.

And note what temperature is not: a correctness dial. Turning it down makes the model more consistent, not more truthful. A confidently wrong answer at temperature zero is still wrong, just reproducibly so.

Because the two phases respond to different inputs, the fixes are different, and it is worth being explicit about which lever moves which number.

To reduce time to first token: shorten the prompt, cache the stable prefix, or reduce queueing. Nothing about the answer affects it.

To reduce total response time: cap output tokens, ask for a more compact format, or stream so the user sees progress while the rest arrives. A smaller model helps here more than anywhere else, since per-token cost falls with weight size.

To reduce cost per token: raise batch size, which is a serving-side concern, and shorten prompts, which is yours. On a per-request basis, prompt tokens usually dominate the bill — a 2,000-token prompt with a 200-token answer is ten to one before any pricing asymmetry.

To reduce tail latency: cap maximum output tokens. The tail here is unusual in that it is partly a property of what the model decided to say, and an unbounded generation is an unbounded latency.

Speculative decoding deserves a mention because it attacks the decode bottleneck directly: a small draft model proposes several tokens, the large model verifies them in one parallel pass, and accepted drafts come for free. Output is identical to normal decoding, so it is a pure latency win where it applies.

A support summariser takes 30 seconds per ticket and the team wants it under 10. Measured breakdown:

queue                              0.4 s
prefill  (6,000-token transcript)  1.1 s
decode   (1,200-token summary)    28.5 s   <- 95%

The instinct is usually to trim the prompt, since 6,000 tokens looks like the big number. It would save about half a second.

Instead, three changes to the output side:

1. Ask for structured bullets, not prose      1,200 -> 320 tokens
2. Cap max_tokens at 400                      hard ceiling on the tail
3. Stream to the UI                           perceived wait = TTFT, ~1.5 s

new decode   320 x 0.012                      3.8 s
new total                                     5.3 s

Under target, with no model change, no hardware change and no serving work. The diagnosis came entirely from knowing which phase owned the time.