Parameters & layers

In one line

A model is a fixed pile of numbers arranged in repeated blocks, and its size tells you its memory footprint and its latency floor directly.

Why it exists

"8B", "70B", "405B" get quoted like clock speeds, and like clock speeds they are a real number attached to a poorly understood meaning. Knowing what a is converts that marketing number into two things you can actually plan with: how much memory you need to hold the model, and how much arithmetic each token costs.

Nothing else about serving a model makes sense until those two are concrete. Why does a 70B model need multiple GPUs? Why does a longer prompt cost more than a longer answer per token? Why does speed things up rather than just shrinking them? All three fall straight out of parameter count and layer structure.

The same block, repeated. Depth is latency and width is memory, and both are fixed the moment the model is trained.

What the numbers actually are#

A parameter is a single learned float. Nothing more. They are grouped into matrices, the matrices into blocks, and the blocks stacked.

Three groups account for essentially all of them:

The — one row per vocabulary entry. At a 128,000-token vocabulary and 4,096 dimensions, that is about 524 million parameters before any processing exists at all. In small models this is a surprisingly large share of the total.

The projections — the query, key, value and output matrices in every block. These are what let tokens exchange information.

The networks — per-token transformations inside every block, typically expanding to four times the model width and back. These are usually about two thirds of all parameters, which is the opposite of most people's intuition, since attention gets all the attention.

Memory follows immediately. At 16-bit , each parameter is two bytes, so an 8B model needs roughly 16GB for weights alone. Add for in-flight requests and activation working space, and a 24GB card is comfortable while a 16GB card is not. A 70B model at the same precision needs about 140GB, which is why it arrives as a multi-GPU deployment or a quantized one.

Why depth costs what it does#

A is one block: attention, then a feed-forward network, each wrapped around a that carries information forward and that every component reads from and adds back into. Models stack 32, 80, 126 of these. Layer 60 is structurally identical to layer 2; only its weights differ.

Depth is strictly sequential. Layer n cannot start until layer n-1 finishes, so latency per token is the sum over all layers, with no parallelism available within a single request. This is why a deeper model is slower per token in a way that more hardware does not fix — you can add GPUs to serve more requests, but not to make one token come out faster.

Width, by contrast, parallelises well. A wider model does more arithmetic per layer, and that arithmetic is matrix multiplication, which is exactly what a GPU is built for. This is the reason model designers tend to grow width before depth when they have the choice.

The consequence for you is that the two headline dials trade differently. A smaller model is cheaper per token and faster per token. A of a large model into a small one — trained to imitate its outputs on your distribution — often lands within a few points of the large model on your task and an order of magnitude cheaper. That is usually a better lever than any serving optimisation.

The math

One forward pass costs roughly two floating-point operations per parameter per token — one multiply and one add, for every weight:

\text{FLOPs} \approx 2 \times N_{\text{params}} \times N_{\text{tokens}}

So an 8B model processing a 2,000-token prompt:

\text{FLOPs} \approx 2 \times 8{\times}10^{9} \times 2000 = 3.2 \times 10^{13}

32 teraFLOPs. A current data-centre GPU does a few hundred teraFLOPs per second at 16-bit precision, so the arithmetic itself is well under a second — often around a tenth of one.

That number is a good model of prompt processing and a bad model of generation. Producing one output token also touches all 8 billion parameters, but to produce a single token, so the machine spends its time moving 16GB of weights from memory rather than doing useful arithmetic. Generation is bound by , not by , which is the single most useful fact in this topic.

Worked example

Sizing a deployment for an 8B model, 16-bit weights, on one 80GB GPU:

weights            8e9 params x 2 bytes          = 16.0 GB
KV cache per req   2 x 32 layers x 8 kv-heads
                   x 128 dim x 4096 tokens
                   x 2 bytes                     =  0.5 GB
activations, workspace, fragmentation            =  4.0 GB (allow)

free for cache     80 - 16 - 4                   = 60.0 GB
concurrent reqs    60 / 0.5                      = 120

About 120 concurrent 4,000-token requests. Now halve the context to 2,000 tokens and the cache per request halves, so the same card holds 240. Or quantize the weights to 8-bit: weights drop to 8GB, freeing another 8GB, and decoding gets faster because there is half as much to move per token.

Notice what did not change: the number of layers, and therefore the latency of a single token. capacity is a memory problem; single-request speed is a depth problem. Different levers.

Gotchas

Reading parameter count as a quality score. Training data, training duration and post-training matter at least as much, and a well-trained 8B model routinely beats an undertrained 70B one on a narrow task. Parameter count predicts cost reliably and capability only loosely.
Budgeting only for weights. Weights are the floor, not the requirement. KV cache grows with concurrency times context length and is what actually runs out first in production. A model that "fits in 16GB" at batch size one will not serve ten users at long context.
Expecting more GPUs to reduce per-token latency. Sharding a model across cards adds communication between layers; it buys capacity and usually costs a little latency. If you need faster tokens, you need a smaller or shallower model, or — not more hardware.
Assuming quantization only saves memory. Because decoding is bandwidth-bound, halving the bytes per weight typically speeds generation noticeably as well. It is usually a throughput optimisation that happens to also fit in less memory.

Mental model

The model is a read-only dataset that has to be scanned end to end to answer any query — like a table with no index, where every lookup is a full scan. The size of the table sets your memory bill, the scan sets your throughput ceiling, and the only way to make an individual scan faster is to have less table. Everything serving-related is a variation on amortising that scan across more work.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Attention