The same question gave two different answers
In one line
The randomness in a model's output is a setting, not a property; for anything you will compare, test or parse, it should be switched off — and knowing where it lives is the whole trick.
The situation
A finance reviewer runs the invoice assistant twice on the same document. The first time it says the total is £840. The second time, £810. Both are confident. The reviewer, reasonably, stops trusting it — and the engineer, reasonably, wonders how a deterministic program produced two results.
Where the dice are#
At every step the model produces — a score for every possible next token — and then something chooses one. That choice is the only random operation in the pipeline, and it is configurable.
rescales the scores before they become probabilities: low values sharpen the distribution toward the top token, high values flatten it. discards the unlikely tail. At temperature zero, the choice is always the highest-scoring token — — and the output becomes a deterministic function of the input and the model version.
In the invoice case, " £840" scored 0.62 and " £810" scored 0.21. At temperature 1, the second answer appears roughly one time in five. At temperature 0, never. The variation was not the model being unsure in some deep sense; it was a dial left at the default.
Where each setting belongs#
Sampling exists for a reason: prose generated greedily is flat and repetitive, and for a writing assistant a moderate temperature is right. But extraction, classification, anything that produces a value code will act on, and anything you will run in an wants greedy. Reproducibility is a feature you can simply turn on.
Two caveats keep this honest. Determinism is a strong tendency rather than a guarantee — floating-point arithmetic across different batch sizes can produce rare divergence, and a provider changing the model behind the same name changes the output too, so pin the version. And turning temperature down makes the model consistent, not correct. If £840 was wrong, temperature zero makes it reliably wrong, which is better for debugging and no better for the reviewer.
The lasting fix for the reviewer's trust is different: have the assistant cite the line items it summed, so the answer can be checked rather than believed.
Questions to ask
- Is temperature set explicitly for this call, or inherited from a default?
- Is this output a value or prose? Values want greedy.
- Is the model version pinned, so "same input" means the same thing next month?
- Can the output be checked against its source, rather than trusted?
Mindset
Randomness in a model's output is a knob with a default, not a fact about models. Know where it is, set it deliberately per use, and remember that consistency and correctness are separate properties.
Where it connects
Phase 2 · 7 min
Prefill & decode
One request is two different workloads with two different bottlenecks, and every latency number you care about belongs to one or the other.
Classifying with small models · Module C · 6 min
Softmax and temperature
Softmax turns scores into probabilities that sum to one; temperature decides how sure those probabilities sound, and calibration makes them honest.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.