hello·ai

Softmax and temperature

AssumesVector space intuitionEncoders vs decoders

In one line

Softmax turns scores into probabilities that sum to one; temperature decides how sure those probabilities sound, and calibration makes them honest.

Prefer small steps? Guided mode walks through the same ideas one at a time.

Why it exists

A classifier ends with one number per label. Picking the largest is easy, and for a demo that is enough. Production needs a second decision: is the largest one large enough to act on, or should this input go to a human?

That second decision needs a number with a meaning — "the model is 80% sure, and when it says 80% it is right about 80% of the time". Raw scores do not have that meaning. Softmax gives them the right shape, and temperature, chosen against labelled data, gives them the right size. Skip either and every threshold you set downstream is a guess.

RAW SCORES0.580.40BillingAccount÷ τ, softmaxτ = 154%46%a slight leanτ = 0.186%14%calibrated (on this data)τ = 0.0299.99%0.01%near certaintySame scores, same winner. Temperature only changes how sure the answer sounds; calibration picks the τ that tells the truth.
The same two scores, 0.58 and 0.40, read at three temperatures. The winner never changes; the claimed confidence goes from a slight lean to near certainty.

Scores rank, they do not promise#

Whatever produces the scores — against a prototype per label, or the of a small classification head on an — the output has the same limits. The numbers put the labels in order. They do not sum to one, they are not bounded the same way across models, and their absolute size means little.

Take a support ticket that scores 0.58 against Billing and 0.40 against Account. Billing ranks first; that part is reliable. But swap the encoder and the same ticket might score 0.81 and 0.77, with the same order and a very different-looking gap. Neither pair is a probability. Reading 0.58 as "58% likely" is the same mistake as comparing timestamps from two machines whose clocks were never synced: the ordering within one source holds, and the values across sources mean nothing.

Softmax: shares that sum to one#

converts a list of scores into shares. Raise ee to each score, then divide each result by the total. For scores 2, 1 and 0 that is 7.39, 2.72 and 1.00, a total of 11.11, and shares of about 67%, 24% and 9%.

Three properties fall out. Every share is positive. The shares sum to one. And the order never changes, because exe^x only grows with xx. The exponent also does something less obvious: shares depend only on the gaps between scores. Add 10 to every score and the shares do not move, which is why softmax does not care what scale a model's scores happen to sit on — only how far apart they are.

That is also its trap. Cosine scores between texts sit in a narrow band, so their gaps are small. The ticket's 0.58 and 0.40 become about 54% and 46%. The arithmetic is right and the reading is honest: on that scale, the model barely prefers Billing. With fourteen labels instead of two, a winner at 0.58 against thirteen others near 0.40 gets only about 8%, while still ranking first.

Temperature: how sure the numbers sound#

is a single number, τ\tau, that every score is divided by before softmax. At τ=1\tau = 1 nothing changes. Below one, gaps grow and the winner's share rises; above one, gaps shrink and shares drift toward even. It is the same knob a chat model exposes for sampling, used here for a different purpose: to set confidence rather than variety.

Temperature never changes the winner — dividing every score by the same positive number keeps the order — so it cannot make a classifier more accurate. What it changes is the claim. At τ=0.02\tau = 0.02 the ticket's 0.18 gap becomes 9, and Billing gets about 99.99%. Same ranking, a far stronger promise.

The right τ\tau is the one that makes the promise true. That property is : among all predictions stated at about 80%, about 80% should be correct. You find it by measurement. Hold out a the model never trained on, try a range of τ\tau values, and for each one compare stated confidence with observed accuracy, band by band. Keep the τ\tau where they agree. Then a threshold on that number can decide whether to answer or to fall back to .

The math

Softmax with temperature, for label ii out of KK with score sis_i:

pi=esi/τ∑k=1Kesk/τp_i = \frac{e^{s_i / \tau}}{\sum_{k=1}^{K} e^{s_k / \tau}}

With two labels only the gap matters. For the ticket, s1−s2=0.18s_1 - s_2 = 0.18, and at τ=0.02\tau = 0.02:

p1=11+e−0.18/0.02=11+e−9≈0.9999p_1 = \frac{1}{1 + e^{-0.18 / 0.02}} = \frac{1}{1 + e^{-9}} \approx 0.9999

When you only have probabilities, not scores, temperature still applies: take the log, divide by τ\tau, exponentiate and renormalise. The first three steps collapse into a power:

eln⁡(p)/τ=p1/τe^{\ln(p) / \tau} = p^{1/\tau}

So at τ=0.5\tau = 0.5, probabilities of 0.7 and 0.3 become 0.49 and 0.09, which renormalise to about 0.84 and 0.16.

Worked example

Two tickets, two labels, a confidence threshold of 80%. Scores are cosine similarities to each label's prototype. The numbers are illustrative.

                          Billing  Account  gap
ticket 1 "charged twice"   0.58     0.40    0.18
ticket 2 "update my card"  0.50     0.45    0.05

Read at three temperatures, the Billing share for each:

             tau = 1    tau = 0.1    tau = 0.02
ticket 1      54%        86%          99.99%
ticket 2      51%        62%          92%

At τ=1\tau = 1 the model never clears 80%, so everything goes to a human and the classifier does no work. At τ=0.02\tau = 0.02 both tickets clear it, including ticket 2, where the model barely leans at all. At τ=0.1\tau = 0.1 ticket 1 is answered and ticket 2 is routed to a person — which is the behaviour you wanted, but only the validation set can tell you that 0.1 is the right value.

Suppose it does: across held-out tickets, the ones stated at about 85% were right about 84% of the time at τ=0.1\tau = 0.1, and the ones stated at 99% were right only about 80% of the time at τ=0.02\tau = 0.02. The first is calibrated; the second is a model promising far more than it delivers. The winner was Billing in every column. Only the decision to answer changed.

Gotchas

  • Reading raw similarity as a percentage. A cosine of 0.9 is an angle, not a chance, and its meaning shifts between encoders. Only a calibrated output can be read as "90% sure", and only for the model and data it was calibrated on.

  • Choosing temperature by feel. A low τ\tau makes outputs look decisive and demos look good. It also turns slight leans into 99% claims, and a confident wrong answer is the most expensive kind. Fit τ\tau on a validation set and refit it when the encoder, the labels or the traffic change.

  • Forgetting to renormalise. ese^{s} for one label, or p1/τp^{1/\tau} without dividing by the new total, is not a probability. Everything downstream — a threshold, a blend of two scorers — silently inherits the error.

  • Comparing top-label shares across tasks. The winner's share falls as the number of labels grows: 54% with two labels can be the same lean as 8% with fourteen. Set thresholds per task, from that task's validation data, never borrowed from another model.

Mental model

Think of a health-check score behind a load balancer. For choosing the best backend, any score works as long as it ranks them — that is softmax on raw scores. For deciding whether to send traffic at all, the score has to mean "the probability this request succeeds", measured against real outcomes, the way an SLO is. Temperature is the calibration that turns the first kind of number into the second. Ranking decides what; calibrated confidence decides whether.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Back to the track

This topic has 3 subtopics.