Why the first word takes a second and the rest take five
In one line
“The model is slow” is two complaints about two phases with two fixes; the split between them is visible in one measurement almost nobody takes.
The situation
Users say the assistant is slow. Engineering measures a median of six seconds per answer and opens a ticket titled "reduce LLM latency". The proposals on it are a smaller model, a faster GPU, and trimming the prompt. All three would help a little. None of them targets the thing the users are actually feeling, because "six seconds" is the sum of two different waits.
Two phases, two clocks#
A request has a phase — the whole prompt processed in one parallel pass — and a phase, one output token per full pass through the model. Prefill scales with prompt length; decode scales with answer length. They are bound by different hardware limits and respond to different levers.
Measure them separately and the six seconds decomposes. Say the prompt is 6,000 tokens and the answer is 400: about 0.4 seconds queueing, 1.1 seconds of prefill, and 4.8 seconds of decode at 12 milliseconds a token.
is 1.5 seconds. That is the wait before anything appears, and it is what a user experiences as "slow to start". The remaining 4.8 seconds is times output length — "slow to finish".
The ticket's three proposals mostly attack prefill. A smaller model helps both phases. Trimming the prompt only shortens the 1.1 seconds. The 4.8 seconds — the larger part — is untouched.
Fixes that match the phase#
If the complaint is really "nothing happens for ages", the levers are prompt length, for the stable prefix, and queueing. Caching alone can take a 6,000-token prompt's prefill from 1.1 seconds to a tenth of that when most of it is a shared system prompt.
If the complaint is "it takes forever to finish", the lever is output length. Ask for a compact format, cap the maximum tokens, and — above all — turn on . Streaming changes nothing about total time and everything about perceived time: the reader starts at 1.5 seconds instead of at 6, and reads faster than the model writes.
The diagnostic habit is the whole lesson. Log time to first token and total time as two numbers. Their ratio tells you which phase to work on before anyone touches a model or a GPU.
Questions to ask
- Do I know time to first token and total time separately, per request?
- Is the prompt or the answer the larger share of the tokens?
- Is the response streamed? If not, that is the first change.
- How much of the prompt is identical across requests and could be cached?
Mindset
"Slow" is not a measurement. Split every model call into the wait before the first token and the wait after it, and the right fix usually names itself.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.