hello·ai
ExampleBest after Phase 24 min

Ops wants the GPU bill halved by Friday

#cost#evals#serving

In one line

Five ways to cut inference cost, each with a different quality price, and only the eval column can tell you which ones you can afford.

The situation

The 13B model serving the internal assistant costs more than planned, and operations asks for the bill halved. The engineer knows four or five ways to do it and has no way to choose between them, because each one changes the model's behaviour and nobody has a number for "behaviour". That number is the whole decision.

modelmemoryspeedevaldo nothing13B · fp1626 GB18 ms/tok91.5%quantize to 8-bit13B · int813 GB13 ms/tok91.2%quantize to 4-bit13B · int47 GB11 ms/tok89.0%smaller model7B · fp1614 GB10 ms/tok88.1%distil to 3B3B · fp166 GB5 ms/tok90.4% on our taskFive ways to cut the bill. Only the eval column tells you which ones you can afford.
Five ways to cut the bill. Only the eval column tells you which ones you can afford.

The menu#

Quantize to 8-bit. Store weights in half the bytes. Memory halves, and because decoding is bound by rather than arithmetic, generation gets faster too. Quality is usually indistinguishable — but "usually" is a measurement, not a promise.

Quantize to 4-bit. A quarter of the memory, faster still. The average quality holds up surprisingly well; the loss concentrates in specific places — multi-step arithmetic, rare names, calibration — that a quick manual check will not sample.

Use a smaller model. A 7B instead of 13B: half the everything, and a capability gap that is real on hard prompts and invisible on easy ones.

Distil. Have the 13B label a few hundred thousand of your real requests, then train a 3B model on them. produces something much better at your task than its size suggests, and no better than that size anywhere else. It needs traffic to distil from, so it is a later-stage option.

Cut the prompt. Not on the menu as proposed, and often the largest lever: if prompts average 3,000 tokens and could be 1,500, the bill halves with no model change at all. Check this first.

The column that decides#

Lay the options out as a table: memory, speed, cost, and a quality score on a of a few hundred real requests. The first three columns you can compute. The fourth requires an to exist, and if one does not, that is the first day's work.

With it, the decision is legible. 8-bit at −0.3 points is free. 4-bit at −2.5 overall but −4.5 on the arithmetic stratum is a product question — does the assistant do arithmetic that matters? The 7B at −3.4 is worse than 4-bit on every axis. The distilled 3B at −1.1 on your task, at a fifth of the cost, is the best option in the table and the one that needed the most preparation.

Without the fourth column every option looks like "cheaper, probably fine", and the one that ships is whichever was easiest, which is how a 4-bit model ends up confidently miscounting refunds.

Questions to ask

  • Is there an eval I trust? If not, that is step one, not step three.
  • Is the eval broken down by task type, or is it one number?
  • Have I checked prompt length before touching the model?
  • Do I have enough logged traffic to distil from yet?

Mindset

Every cost lever below "shorten the prompt" has a quality price. The eval is what converts "probably fine" into a number you can put in a table next to the savings — and a cost decision made without that column is a quality decision made by accident.

Where it connects

Got the shape of it?

Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.

Next: The same question gave two different answers