Cutting the bill by 80% without touching the model
In one line
The proposal was to reserve GPUs; the actual fix was three prompt changes and a router, applied in order of cost, and the GPUs were never needed.
The situation
A document assistant costs $18,400 a month at half a million requests, with a p99 of 22 seconds. The proposal on the table is to self-host on reserved GPUs — a quarter's work and a large commitment — projected to cut the bill by half.
The engineer asked to review it pulls the numbers first: 6,200 prompt tokens per request, 900 output tokens, no caching, every request to the largest model. The proposal is aiming a big lever at a small share of the problem.
Cheapest levers first#
Send fewer passages. Twenty retrieved chunks went into every prompt. the twenty and sending five drops the prompt to 3,400 tokens. Cost falls 26% — and accuracy rises two points, because five good passages beat twenty mixed ones. One day.
Reorder for caching. The system prompt and tool definitions were placed after the retrieved passages, so nothing was reusable across calls. Moved to the front, 2,100 tokens become a stable prefix served by at a fraction of the price. Another 25% off, and time to first token drops from 1.8 to 0.5 seconds. One line in a template.
Cap the output. Answers ran to 900 tokens of prose. Asking for bullets and capping at 400 brings the average to 380 and — because here is set by the longest answers — the p99 from 20 to 9 seconds. Cost down 43% from the previous step. An afternoon.
Three changes, one day, and the bill is at $5,900 — already past the proposal's target, with quality up and latency down.
Route. The fourth step needs a week: a small classifier in front that sends the 61% of requests that are simple lookups to a model a twentieth of the price. is a classification problem with a clean , and it brings the bill to $2,450. Accuracy dips one point on the routed slice, which is a decision the eval makes visible.
What the ordering protects you from#
The GPU proposal was not wrong about the direction; it was wrong about the order. Self-hosting attacks the per-token price. The first three steps attack the number of tokens, which was the larger term, and they were reversible, cheap and fast. Had the reservation gone ahead, the same three prompt changes would still have been the largest win — on top of a quarter's infrastructure work aimed at the smaller term.
The discipline is the same as any performance work: measure where the cost is, fix the largest term with the cheapest change, re-measure, repeat. The expensive move is last, if it is still needed.
Questions to ask
- What share of the cost is prompt tokens versus output tokens versus per-token price?
- Which of the free levers — fewer passages, caching order, output cap — have been tried?
- What fraction of requests could a much smaller model handle, and how would I know?
- After each change, did quality move — and did I check?
Mindset
Profile before proposing. Most of an LLM bill is tokens, most tokens are prompt, and most prompts are longer than they need to be. The cheap levers come first, and the infrastructure project is what you do if they run out.
Where it connects
Phase 3 · 6 min
Context engineering
Deciding what occupies a finite window, in what order, on every call — and it is mostly a discipline of leaving things out.
Phase 4 · 6 min
Workflow patterns
Four shapes for composing model calls, ordered by how much control flow you hand over — and the right default is the one that hands over the least.
Phase 4 · 6 min
Cost & latency levers
Six levers, three of which need no infrastructure at all and are almost always tried last.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.