The adaptation ladder

1. Prompt and instructions. Rewrite the system prompt, be specific about format and constraints, state what to do when the answer is unknown. Minutes to change, instant to revert, zero marginal cost. It is astonishing how often this is the whole answer, and equally astonishing how often it is skipped in favour of something that sounds more like engineering.

2. Few-shot examples. Two to five worked examples in the prompt. They pin down output shape and edge-case handling far more reliably than a paragraph of description, and they cost tokens on every call — though those tokens sit in the cacheable prefix, so the real cost is lower than it looks. Combined with structured output, this rung handles most formatting problems outright.

3. Retrieval. Put the relevant facts in the context at request time. This is the correct rung for anything the model does not know, and the only one where a fact updated in your database is live on the very next request. Days of work for the first version.

4. LoRA adapters. Freeze the base weights, train a small pair of low-rank matrices alongside. Megabytes rather than gigabytes, several adapters can be served over one base model, and quality approaches full fine-tuning for style and format tasks. Weeks of work, and now you own a training pipeline and a dataset.

5. Full fine-tuning or Distillation. Update every weight, or train a smaller model to imitate a larger one on your traffic. Months, real money, and a permanent operational commitment — but distillation in particular can cut inference cost by an order of magnitude on a narrow task, and it is the one rung that improves Throughput rather than just quality.

The choice is not about the size of the problem. It is about what kind of problem it is.

Wrong facts → retrieval, every time. Fine-tuning teaches facts poorly, locks them to a training snapshot, and produces a model that states last quarter's numbers with total confidence.

Wrong format or tone → examples first, then LoRA if examples are not enough. This is the case fine-tuning is genuinely good at, because form is exactly what gradient descent on your outputs teaches.

Wrong reasoning on a specialised domain → try retrieval of worked examples first, then a stronger model, then fine-tuning. The middle option is the one people skip.

Too slow or too expensive → this is the distillation case, and the only one where climbing is about economics rather than quality. It requires production traffic to distil from, so it is structurally a later-stage move.

The asymmetry that should drive the decision is reversibility. Rungs 1 to 3 are config changes: reverted in a deploy, with no artefacts left behind. Rungs 4 and 5 create a dataset, a training pipeline, a model artefact and a versioning problem that outlive whoever built them. That gap is much larger than the compute cost, and it is the part that does not show up in the estimate.

Climbing without measurement is how teams end up with an expensive pipeline that underperforms a prompt they wrote in an afternoon. The rule is simple: you need a golden set and a baseline before you climb, and the same Eval run after.

Two failure patterns worth naming:

Climbing past the ceiling. If retrieval took you from 61% to 89% and fine-tuning takes you to 90%, the fine-tune is not worth a permanent pipeline. Diminishing returns are the signal to stop, and they are only visible with a before-and-after number.

Climbing to fix the wrong layer. A fine-tune will not fix a retrieval recall problem, and no amount of adapter training compensates for chunks that never reach the prompt. Diagnose which stage is failing before choosing a rung — the stratified eval from the previous topic is exactly the tool for this.

One more thing the measurement protects you from: Drift. A prompt-level system inherits provider improvements for free. A fine-tuned model is pinned to a base checkpoint, and upgrading means retraining and revalidating. Climbing converts a free upgrade path into a recurring project.

A legal-document classifier, 14 categories, needs 95% accuracy. Baseline with a generic prompt: 72%.

rung                                          accuracy   effort    reversible
-------------------------------------------------------------------------
0  generic prompt                                72%        —          —
1  rewrite: define each category precisely       81%      2 hours     yes
2  add 5 few-shot examples + enum schema         88%      1 day       yes
3  retrieve 3 similar labelled past docs         93%      4 days      yes
4  LoRA on 4,000 labelled documents              96%      3 weeks     no
5  full fine-tune on 40,000 documents            96.4%    2 months    no

Rung 4 hits the target. Rung 5 adds 0.4 points for six times the effort and a much larger dataset requirement — a clear stop.

But look at the shape of the first three rungs: 72% to 93% for five days of work, all of it reversible, none of it creating an artefact anyone has to maintain. Two of those points came from writing down what the categories actually mean, which is not machine learning at all.

The honest read of the table is that rung 4 is justified here — 93% to 96% matters when the alternative is manual review — and that it would not have been justified if the target had been 92%. That is the decision the ladder exists to make legible.