Softmax

The step that turns a pile of raw scores into percentages that add up to 100.

Part of the Turning the dials track on lAItest.

Raw scores can be negative, can be enormous, and add up to nothing useful.

Softmax is the repair job.

Softmax turns a list of scores into percentages that add to 100.

It does two things. First it exaggerates: each score is fed through an exponential, so a score that is slightly higher becomes a probability that is a lot higher. Then it divides everything by the total, so the list sums to one. The ordering never changes — the top-scoring token stays the most likely one. Only the gaps between them change.

Try it

Watch a scoreboard become percentages. When one score leads by a lot, notice how little is left for everyone else. This step is an interactive widget; open the lesson to use it.

A common misconception

Commonly believed: The percentage next to a token is how confident the model is that it is correct.

Actually: It is an estimate of how often that token would come next in text like the text it learned from. A token at 90% is not a fact verified nine times out of ten. Fluent and wrong scores exactly as high as fluent and right, because the scoreboard was never measuring truth.

In one sentence

Softmax converts scores into odds, and the exponential inside it is why a small lead turns into a landslide.