The flaky test that isn't flaky
In one line
A test that passes four times in five is not broken — it was written for a function and the thing under test is a distribution, and every habit you have about testing needs one amendment.
The situation
An engineer adds the first test for the new summariser: feed it a fixed
support transcript, assert the summary equals the expected string. It passes.
It passes again. On the third CI run it fails — the summary is fine, but
worded differently. The test gets tagged @flaky, then retried three times
automatically, then quietly deleted.
Nothing was flaky. A deterministic function was being asserted on, and the component is not one. This example needs no knowledge of how a model works — only a willingness to reconsider what a test is.
What changed about the contract#
Every test you have written assumes that f(x) is the same value every time.
That assumption is so deep it is invisible, and a model call breaks it in two
ways.
First, the output is sampled. Unless you set to zero and use , the same input yields different outputs by design. That much you can switch off, and for testing you should.
Second — and this is the part that survives switching randomness off — the output is free text, and correctness is a property of its meaning rather than its bytes. Two summaries with different wording can both be right. An exact-match assertion encodes one of them as the only acceptable answer, which is not what anyone believes.
So the contract is not "returns this string". It is "returns something with these properties, most of the time". Once that is said out loud, the test becomes writable again.
Rewriting the test#
Replace the single assertion with properties: the summary mentions the order number; it is under 80 words; it does not contain any phrase from the forbidden-promises list; it names the resolution the transcript actually reached. Each of those is a boolean you can check in code, on every run, and none of them cares about wording.
Then replace the single input with fifty — real transcripts, with the properties labelled by hand. That is a , and the test's output is now a percentage: 47 of 50 pass. The assertion becomes a threshold: fail the build if it drops below 44, or if it drops by more than three from the last release.
That is an . Structurally it is a test suite with two amendments — assertions are properties, and the pass condition is a rate — and everything else you know about testing transfers: fixtures, CI gates, a case for every bug, fast feedback.
The math
With n cases and a true pass rate p, the observed rate has a standard error of
At p = 0.9 and n = 50:
About four points of noise. A gate set two points below the baseline will fail on noise alone; one set five points below will not. Fifty cases is enough to catch a collapse and not enough to detect a small regression — which is worth knowing before anyone celebrates a two-point improvement.
Questions to ask
- What properties must every acceptable output have? Those are the assertions.
- How many real examples do I have with those properties labelled?
- What pass rate would I actually accept, and how much noise is in that number?
- Is randomness switched off for this test? If not, why not?
Mindset
A model is a component with a statistical contract. Do not try to make it behave like a function; change what the test asserts. Properties instead of strings, a rate instead of a boolean, and a set of real cases that grows with every failure. The discipline is identical — only the assertion changed shape.
Goes deeper in
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.