Quantization

In one line

Storing weights in fewer bits to cut memory and bandwidth, which usually makes generation faster as well as smaller — and moves quality loss into the cases you are least likely to test.

Why it exists

An 8B model at 16-bit precision needs 16GB just for weights. That does not fit on a consumer GPU, and on a data-centre card it leaves less room for , which caps how many users you can serve.

stores each in fewer bits — 8 or 4 instead of 16. The immediate effect is proportionally less memory. The more interesting effect follows from where decoding spends its time: since every reads the whole model, halving the bytes per weight halves the memory traffic per token, and because decoding is bound by rather than arithmetic, that translates into real speed.

It is one of the few optimisations that improves size, cost and latency at once. The price is paid in quality, in a distribution that is easy to miss.

Fewer bits per weight means less memory to move per token, so quantizing is usually a speed change as much as a size change.

What is actually being thrown away#

A 16-bit float represents a wide range of values with fine gradations. An 8-bit integer has 256 distinct values. Quantization maps a range of floats onto that smaller set and stores a scale factor to map back.

Done naively — one scale for the entire model — this is a disaster, because weight distributions differ by orders of magnitude between layers. Done in practice, the scale is per-tensor or per-channel, and sometimes per small block of values, so each group gets a range fitted to its own distribution.

The distinction that matters most is what is quantized:

Weights only is the common case. Weights are static, so they can be quantized once, offline, with careful calibration. This is where nearly all of the memory and bandwidth benefit comes from.

Activations are the values flowing between layers. They are dynamic and contain outliers — a few dimensions with values far outside the typical range — and those outliers carry disproportionate information. Quantizing them is harder and gives less benefit.

The KV cache can be quantized independently of the weights, typically to 8 bits, and this is often the highest-leverage version for a long-context workload, because it is the cache and not the weights that limits concurrency.

Methods vary in how much calibration data they use and whether they reconstruct layer outputs or just round — GPTQ and AWQ are the common weight-only ones, and the choice between them matters less than the bit width.

What it costs, and where#

loss is not uniform. Average behaviour barely moves; the tails move a lot. This is the part that catches people.

At 8-bit weights, quality is close to indistinguishable for most workloads. moves by a fraction of a percent, and task evals typically land within noise. It is close to a free win.

At 4-bit, the picture is more interesting. Average scores still hold up remarkably well, and the degradation concentrates in exactly the places you would not sample: long multi-step arithmetic, rare entity names, code with unusual identifiers, and the model's calibration — its sense of when it is unsure. A 4-bit model tends to be slightly more confident and slightly more wrong at the same time.

There is also a scale interaction worth knowing: larger models tolerate quantization better than small ones. A 70B model at 4-bit is often stronger than a 13B model at 16-bit using the same memory, which makes "quantize a bigger model" a real strategy rather than a compromise.

The thing that makes this risky in practice is the shape of the loss. Because the average is fine, a quick manual check will pass. The regression shows up in production as a slightly higher rate of subtle errors — which is precisely the signal an with a proper is built to catch and that eyeballing twenty outputs is not.

The math

Quantization maps a float range onto integers with a scale and a zero point. For symmetric 8-bit quantization of a tensor whose values span $[-w_{max}, w_{max}]$ :

q = \text{round}\!\left(\frac{w}{s}\right), \qquad s = \frac{w_{max}}{127}

Take a weight tensor with $w_{max} = 0.5$ , so $s = 0.5 / 127 = 0.00394$ . A weight of $0.3172$ :

q = \text{round}\!\left(\frac{0.3172}{0.00394}\right) = \text{round}(80.5) = 80

Dequantized back:

\hat{w} = 80 \times 0.00394 = 0.3152

An error of 0.002, about 0.6%. Small per weight, and it accumulates across billions of them and dozens of layers — which is why the aggregate effect is measurable even though each rounding looks negligible.

Worked example

A 13B model, currently 16-bit on an A100 80GB, serving 4,000-token contexts. Measured: 26GB weights, 42 concurrent requests, 18ms per output token.

Quantize weights to 4-bit:

weights           26 GB -> 7 GB          (19 GB freed)
cache available   50 GB -> 69 GB
concurrency       42 -> 58 requests      (+38%)
per-token time    18ms -> 11ms           (-39%, less to move per token)

Then run the golden set, 200 labelled production queries:

16-bit baseline      91.5% correct
4-bit                89.0% correct        -2.5 points
4-bit, errors by category:
  factual recall     -1.0
  multi-step maths   -4.5   <- concentrated here
  formatting         -0.2

The aggregate 2.5-point drop looks acceptable. The 4.5-point drop on multi-step arithmetic might not be, depending on the product. That breakdown is only visible because the eval was segmented — an overall score would have shown 89% and hidden the actual decision.

The sensible landing point here is often 8-bit: most of the memory win, most of the speed win, and a quality delta inside the noise.

Gotchas

Validating on aggregate scores only. The loss concentrates in specific capabilities, so a single number hides it. Segment your eval by task type before and after, and look at the worst segment rather than the mean.
Assuming the speedup follows the size reduction. It does at low , where you are bandwidth-bound. At high batch sizes the workload becomes compute-bound and quantized gains shrink or vanish. Measure at your real concurrency, not at batch size one.
Quantizing weights and forgetting the cache. For long-context workloads the KV cache is the binding constraint, and 8-bit cache often frees more useful memory than 4-bit weights do. They are independent settings; tune them separately.
Comparing quantized checkpoints from different sources. "4-bit" says nothing about group size, calibration data or which layers were kept wide. Two 4-bit builds of the same model can differ by several points. Pin the exact artefact, not the bit width.
Reaching for it before trying a smaller model. into a genuinely smaller model, or simply using a smaller one, often beats a quantized large model on both cost and latency for a narrow task. Quantization is the right answer when you need the large model's breadth.

Mental model

It is lossy compression on the weights, and it behaves like every other lossy compression you have shipped: nearly imperceptible at moderate settings, degrading gracefully, and failing first on the content furthest from the typical case. You would not ship a new JPEG quality setting without looking at the images that compress worst. Same discipline, same reason.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Ops wants the GPU bill halved by Friday
Five ways to cut inference cost, each with a different quality price, and only the eval column can tell you which ones you can afford.
after phase 2 · 4 min · #cost #evals #serving

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Retrieval & RAG