The same question gave two different answers
In one line
The randomness in a model's output is a setting, not a property; for anything you will compare, test or parse, it should be switched off — and knowing where it lives is the whole trick.
The situation
A finance reviewer runs the invoice assistant twice on the same document. The first time it says the total is £840. The second time, £810. Both are confident. The reviewer, reasonably, stops trusting it — and the engineer, reasonably, wonders how a deterministic program produced two results.
Where the dice are#
At every step the model produces — a score for every possible next token — and then something chooses one. That choice is the only random operation in the pipeline, and it is configurable.
rescales the scores before they become probabilities: low values sharpen the distribution toward the top token, high values flatten it. discards the unlikely tail. At temperature zero, the choice is always the highest-scoring token — — and the output becomes a deterministic function of the input and the model version.
In the invoice case, " £840" scored 0.62 and " £810" scored 0.21. At temperature 1, the second answer appears roughly one time in five. At temperature 0, never. The variation was not the model being unsure in some deep sense; it was a dial left at the default.
Where each setting belongs#
Sampling exists for a reason: prose generated greedily is flat and repetitive, and for a writing assistant a moderate temperature is right. But extraction, classification, anything that produces a value code will act on, and anything you will run in an wants greedy. Reproducibility is a feature you can simply turn on.
Two caveats keep this honest. Determinism is a strong tendency rather than a guarantee — floating-point arithmetic across different batch sizes can produce rare divergence, and a provider changing the model behind the same name changes the output too, so pin the version. And turning temperature down makes the model consistent, not correct. If £840 was wrong, temperature zero makes it reliably wrong, which is better for debugging and no better for the reviewer.
The lasting fix for the reviewer's trust is different: have the assistant cite the line items it summed, so the answer can be checked rather than believed.
Questions to ask
- Is temperature set explicitly for this call, or inherited from a default?
- Is this output a value or prose? Values want greedy.
- Is the model version pinned, so "same input" means the same thing next month?
- Can the output be checked against its source, rather than trusted?
Mindset
Randomness in a model's output is a knob with a default, not a fact about models. Know where it is, set it deliberately per use, and remember that consistency and correctness are separate properties.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.