Evals

In one line

Automated tests for a component whose output is not stable, where the assertion is a threshold on a score and the suite is the only thing making changes measurable.

Why it exists

You change a prompt. Is it better? Without measurement the honest answer is that you have no idea, because the same input produces different outputs run to run, and the twelve examples you tried by hand are not a sample.

This is the uncomfortable part of building on a model: you have introduced a component with no deterministic contract into a codebase where every other component has one. An restores the contract at the level it can exist — a statistical one over a fixed set of cases.

Teams that skip this do not move slowly. They move fast for two months and then stop being able to change anything, because every change is an unfalsifiable claim.

The score fluctuates because the system is non-deterministic, so the gate is a threshold and a trend — never an exact match.

The golden set is the whole asset#

An eval is inputs, expected outputs and a grader. The inputs are the part that matters, and they should come from real traffic.

A of fifty to two hundred cases drawn from production, including every failure that has already happened once, finds more regressions than a thousand synthetic ones. Synthetic cases test what you imagined; real ones test what users do, and the gap between those is where the bugs are.

Build it incrementally, and make one rule non-negotiable: every production failure becomes a case. That single habit converts your eval suite from a static artefact into something that compounds. It is the same discipline as writing a regression test for every bug, and it works for the same reason.

Stratify it. An overall score of 87% tells you nothing about which capability moved. Tag each case — extraction, refusal, multi-turn, long-context, adversarial — and report per stratum. Almost every interesting regression is concentrated in one tag and invisible in the mean.

Include cases that should fail: out-of-scope questions where the right answer is a refusal, inputs with no answer in the corpus where the right answer is "I don't know", and attempts. A system measured only on questions it can answer will be optimised into one that answers everything.

Grading, from cheapest to loosest#

Exact match works when there is a right answer: a classification label, an extracted id, a number. Cheap, deterministic, unambiguous. Use it everywhere it applies, which is more places than people assume once is in play.

Programmatic checks cover properties rather than values: is it valid JSON, does every cited chunk id appear in what was retrieved, is it under the length limit, does it avoid a forbidden phrase. These are ordinary assertions and they catch a surprising share of real regressions.

Retrieval metrics are separate and often skipped. at k against labelled relevant documents measures the search half independently of the generation half, and when end-to-end quality drops this is the first number to check.

handles open-ended output where no exact answer exists. It scales, and it brings its own biases: toward longer answers, toward its own phrasing, toward the first option in a pair. Mitigate by grading against a written rubric rather than a vibe, by scoring one dimension at a time, by randomising position in pairwise comparisons, and — the step that makes it trustworthy — by calibrating against a few dozen human labels and reporting the agreement rate. An uncalibrated judge is a number with unknown units.

Human review stays necessary for the top of the funnel. Twenty cases a week, by someone who knows the domain, catches things no automated grader will.

Running it like a test suite#

Non-determinism changes the mechanics, not the principle.

Pin everything. Model version, temperature, prompt template, retrieval index snapshot. Set to zero and use so run-to-run variance is minimised — you want changes attributable to your change, not to sampling.

Run n times where variance matters. For anything sampled, three runs and a mean is a better signal than one run, and the spread itself is informative.

Gate on a threshold and a trend, not an exact value. "Overall ≥ 85% and no stratum down more than 3 points from the last release" is a workable gate. An exact-match gate on a non-deterministic system is a flaky test, and a flaky gate gets disabled within a month — which is worse than having no gate.

Run on a schedule as well as on deploy. is real: providers update models, your corpus grows, user phrasing shifts. A nightly run against a pinned suite is how you notice that your code did not change and your behaviour did.

Track cost and latency alongside quality. A prompt change that gains two points and doubles the token count is a decision, not an improvement, and it should be visible as one.

The math

Scores move run to run, so the question is whether a difference is real. For two proportions over the same n cases, the standard error of the difference is roughly:

SE \approx \sqrt{\frac{p_1(1-p_1) + p_2(1-p_2)}{n}}

With 100 cases, a baseline of 0.85 and a new score of 0.88:

SE \approx \sqrt{\frac{0.85(0.15) + 0.88(0.12)}{100}} = \sqrt{\frac{0.233}{100}} = 0.048

The observed gain is 0.03 against a standard error of 0.048. That is noise. At 100 cases you can reliably detect differences of roughly ten points and nothing smaller — which is worth knowing before you ship a three-point "improvement".

Want to detect five points? You need roughly four times the cases. This is the argument for growing the golden set, and the reason a 20-case suite tells you almost nothing.

Worked example

A document Q&A system, 150 golden cases, stratified:

stratum          n     baseline
simple lookup    60      94%
multi-hop        35      71%
no answer        25      88%    (correct refusal)
adversarial      20      95%    (resisted injection)
long context     10      60%
                ---     ----
overall         150      85%

A prompt change intended to reduce hedging:

stratum          baseline   new     delta
simple lookup       94%     96%      +2
multi-hop           71%     74%      +3
no answer           88%     52%     -36    <-
adversarial         95%     91%      -4
long context        60%     62%      +2
                   ----    ----
overall             85%     83%      -2

Overall moved two points, which looks like noise and would pass a loose gate. The stratified view shows what actually happened: telling the model to hedge less destroyed its willingness to say "I don't know". Refusal accuracy fell by thirty-six points.

That regression is the single most damaging one this system could have, it went live on a change that read as a small copy edit, and only the stratification made it visible. This is why per-stratum reporting is not a refinement.

Gotchas

Reporting only an overall score. Regressions concentrate in one capability and average away to nothing. Stratify, and gate per stratum.
Building the golden set from imagination. Synthetic cases test what you thought of. Production traffic tests what users do. Sample from logs, and add every real failure as a case the day it happens.
Trusting an uncalibrated judge. Model graders have systematic biases. Score a few dozen cases by hand, measure agreement, and report that number alongside the eval — otherwise the scale is unknown.
Omitting cases that should fail. Measure refusals, unanswerable questions and adversarial inputs, or you will optimise toward a system that answers everything confidently.
Running only on deploy, against too few cases. Twenty cases cannot distinguish an improvement from a coin flip, and behaviour drifts while your code sits still. A scheduled run over a growing set is your for that, and it is what tells you when climbing the ladder has stopped paying.

Mental model

It is a for a component with a probabilistic contract. Everything you know about testing transfers — fixtures, CI gates, a test per bug, fast feedback — with two amendments. Assertions become thresholds, and a failing run means "investigate the distribution" rather than "this line is wrong". The discipline is identical; only the assertion changed shape.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: The adaptation ladder