hello·ai
Phase 4 · Systems6 min

Cost & latency levers

AssumesPrefill & decodeContext engineering

In one line

Six levers, three of which need no infrastructure at all and are almost always tried last.

Why it exists

An LLM feature that works in a demo and costs $40,000 a month in production is a common outcome, and it is usually fixable without changing the model or the architecture. The levers are well understood; the problem is that they are applied in the wrong order.

The order that matters is by cost of implementation, not by size of effect. Three of the six levers are prompt-level changes that ship in an afternoon, and they are routinely skipped in favour of infrastructure work that takes a month.

shorten the prompt-40% input costfreecache the stable prefix-70% TTFTfreecap max output tokensbounds the p99freeroute easy work to a small model-80% on that slicea router + evalsraise batch size+throughputlatency, if overdonequantize-50% memorya quality checkgainscosts youThe shaded three need no infrastructure and are almost always tried last.
The three free levers at the top need no new infrastructure, and are almost always reached for after the three below them.

Know which number you are moving#

Before pulling anything, be precise about the target, because the levers do not overlap as much as people assume.

Cost per request is dominated by prompt for most applications — often ten to one against output tokens before any pricing asymmetry. If you have not measured the split, measure it first; a surprising number of teams optimise output length on a workload that is 90% input.

is prefill plus queueing. Output length has no effect on it whatsoever. This is the number a user in a streaming interface experiences as responsiveness.

Total response time is time to first token plus output tokens times . On a long answer, the second term dominates by an order of magnitude.

is unusual here: it is partly a property of how verbose the model decided to be, which means a hard cap on output tokens is the most direct control you have over p99.

The three free levers#

Shorten the prompt. The most direct cost reduction available and the most frequently ignored. Rerank retrieved passages and send five instead of twenty. Summarise conversation history beyond six turns. Delete the instructions that accumulated during prompt iteration and no longer earn their tokens. Halving a 4,000-token prompt halves the dominant cost immediately, and — as showed — accuracy frequently rises, because fewer, better passages beat more, worse ones.

Cache the stable prefix. means a repeated prefix skips prefill entirely. Providers charge a fraction of the normal input price for cached tokens, and time to first token drops sharply. It requires only that you order the prompt stable-first, volatile-last — which costs nothing and is a one-line change to a template.

Cap output tokens. Sets a ceiling on the slowest and most expensive part of the request, bounds your p99, and bounds your worst-case bill. Combine it with asking for a compact format: bullets instead of prose typically cuts output length by two thirds with no loss of information.

None of these require a deploy of anything but a prompt. All three should be done before anything below.

The three that cost something#

Route to a smaller model. A classifier in front of your pipeline sending easy traffic to a model a twentieth of the price is the largest single cost change available on most systems. It costs you a router, an for the router, and a fallback path when confidence is low.

Raise / use better serving. Continuous batching raises several-fold on self-hosted inference. Irrelevant if you are using a provider API, since it is their concern, and decisive if you are not.

or . Quantization cuts memory and speeds decoding for a measurable quality cost. Distillation — training a small model to imitate a large one on your traffic — can cut inference cost by an order of magnitude on a narrow task. Both need an eval to justify, and distillation needs enough production traffic to distil from, so it is structurally a later move.

A note on , which is often proposed at this point. Serving a stored answer when a new question is close enough in embedding space has an attractive hit rate and a severe failure mode: "close enough" is a threshold, and two questions can be nearly identical in wording with opposite correct answers. Treat it as a correctness decision that happens to improve latency, not the reverse. Exact-match caching on normalised inputs has none of these problems and is usually where the real hits are anyway.

The math

Cost per request, with cached and uncached input priced separately:

C=Pnew cin+Pcached ccached+O coutC = P_{\text{new}} \, c_{\text{in}} + P_{\text{cached}} \, c_{\text{cached}} + O \, c_{\text{out}}

Take 4,000 prompt tokens, 500 output tokens, at 3permillioninput,3 per million input, 0.30 per million cached input and $15 per million output:

C=4000(3×10−6)+0+500(15×10−6)=0.012+0.0075=$0.0195C = 4000(3{\times}10^{-6}) + 0 + 500(15{\times}10^{-6}) = 0.012 + 0.0075 = \$0.0195

Now move 3,000 of those prompt tokens into a cached prefix:

C=1000(3×10−6)+3000(0.3×10−6)+500(15×10−6)C = 1000(3{\times}10^{-6}) + 3000(0.3{\times}10^{-6}) + 500(15{\times}10^{-6}) C=0.003+0.0009+0.0075=$0.0114C = 0.003 + 0.0009 + 0.0075 = \$0.0114

A 42% reduction from reordering a prompt. No model change, no infrastructure, no quality risk.

Worked example

A document assistant at 500,000 requests a month, $18,400 monthly, p99 of 22 seconds. Applied in order:

starting point
  prompt 6,200 tokens · output 900 tokens · no caching
  cost/request $0.0321            p99 22.0 s

1. rerank passages, 20 -> 5           prompt 6,200 -> 3,400
   cost/request $0.0237  (-26%)       p99 20.5 s
   accuracy 86% -> 88%

2. reorder prompt, enable caching     2,100 tokens cached
   cost/request $0.0180  (-25%)       TTFT 1.8 s -> 0.5 s

3. cap output at 400, ask for bullets output 900 -> 380
   cost/request $0.0102  (-43%)       p99 20.5 s -> 9.1 s

4. route: 61% of traffic to a small model
   cost/request $0.0049  (-52%)       p99 9.1 s -> 7.4 s
   accuracy 88% -> 87%

monthly cost   $18,400 -> $2,450   (-87%)
p99             22.0 s -> 7.4 s    (-66%)
accuracy           86% -> 87%

Steps 1 to 3 took a day and delivered a 68% reduction on their own. Step 4 took a week. The infrastructure work that was originally proposed — self-hosting on reserved GPUs — was never needed, and would have been a much larger project aimed at a smaller share of the bill.

Gotchas

  • Optimising output length on an input-heavy workload. Measure the token split before choosing a lever. A workload that is 90% prompt tokens will not notice a shorter answer.

  • Adding infrastructure before shortening the prompt. Self-hosting, custom serving and GPU reservations are large projects. They are the right answer occasionally and the first answer almost never.

  • Leaving max_tokens unset. Your p99 latency and worst-case cost are then decided by the model's verbosity rather than by you.

  • Semantic caching without measuring the false-hit rate. A wrong cached answer is a correctness incident, not a performance regression. Measure how often near-duplicate questions have different correct answers in your domain before enabling it.

  • Changing the model without rerunning evals. Every cost lever below the top three has a quality cost. Measure it deliberately rather than discovering it from support tickets.

Mental model

It is ordinary performance work, and it obeys the same discipline: profile first, then fix the largest term, then re-measure. Shortening the prompt is removing an N+1 query. Prefix caching is an HTTP cache header. Routing to a smaller model is picking the right instance size. Nobody rewrites a service in a faster language before checking whether the query had an index, and nobody should reserve GPUs before checking whether the prompt had twenty passages in it.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Observability

This topic has 3 subtopics.