Phase 2 of 4
The model
What is actually happening inside, and what does that cost?
A transformer is a fixed amount of arithmetic per token. Once you can count that arithmetic, latency, memory and price stop being mysterious.
Topics in this phase
Parameters & layers
6 minUnreadA model is a fixed pile of numbers arranged in repeated blocks, and its size tells you its memory footprint and its latency floor directly.
After Embeddings
Attention
7 minUnreadA soft lookup where every token asks every earlier token how relevant it is, and the answer is a set of weights the model learned rather than a rule anyone wrote.
After Parameters & layers
Training vs inference
6 minUnreadTraining is a write path you will almost certainly never run; inference is a stateless read path, and conflating the two is the source of most bad architecture decisions here.
After Parameters & layers
Prefill & decode
7 minUnreadOne request is two different workloads with two different bottlenecks, and every latency number you care about belongs to one or the other.
After Parameters & layers, Attention
KV cache
6 minUnreadPer-request memory that turns quadratic regeneration into linear growth, and then becomes the thing that runs out first.
After Attention, Prefill & decode
Quantization
6 minUnreadStoring weights in fewer bits to cut memory and bandwidth, which usually makes generation faster as well as smaller — and moves quality loss into the cases you are least likely to test.
After Parameters & layers
Examples that open after this phase
Problem statements that make sense once these topics are done. They unlock at the same 80% mark the next phase does.
Why the first word takes a second and the rest take five
“The model is slow” is two complaints about two phases with two fixes; the split between them is visible in one measurement almost nobody takes.
after phase 2 · 4 min · #latency #debugging
How many users fit on one GPU?
The weights are the fixed cost and the KV cache is the variable one; capacity is a memory calculation you can do on a napkin, and the answer changes by 16x with context length.
after phase 2 · 4 min · #capacity #memory #serving
Summarising a 400-page PDF
The window is large enough to hold the whole document; putting it there is the expensive, lossy, unrepeatable option, and the alternative is a map-reduce you already know how to build.
after phase 2 · 4 min · #context #architecture #cost
The bot forgot my name
Inference is stateless; what feels like memory is the client re-sending the transcript, and the moment the transcript is trimmed to fit, the earliest facts are the first to go.
after phase 2 · 3 min · #memory #context #product
Ops wants the GPU bill halved by Friday
Five ways to cut inference cost, each with a different quality price, and only the eval column can tell you which ones you can afford.
after phase 2 · 4 min · #cost #evals #serving
The same question gave two different answers
The randomness in a model's output is a setting, not a property; for anything you will compare, test or parse, it should be switched off — and knowing where it lives is the whole trick.
after phase 2 · 3 min · #testing #evals #debugging