Softmax: how raw AI scores become chances that add up to 100%

What softmax is

When a model has to choose between several possible answers, it first gives each one a raw score. These scores (called logits, which have their own video in this series) can be any number: negative, tiny or enormous. They are not chances yet, so you cannot read them as "how likely".

Softmax is the step that turns those scores into probabilities: numbers between 0% and 100% that always add up to exactly 100%. After softmax, the model can say something you can actually use, such as "67% this, 24% that, 9% the other".

What the video shows

The video shows softmax splitting 100% among several scores, with the leader taking an outsized share, more than its lead seems to deserve. It also shows a rule that is easy to miss: only the gaps between the scores matter, not how big the scores are.

An everyday example

Imagine a music app guessing which of three moods you are in: calm, upbeat or sad. Its raw scores are 2, 3 and 1. Softmax turns them into roughly 24%, 67% and 9%. Upbeat is only one point ahead, yet it takes two-thirds of the pie.

Now imagine the scores were 800, 801 and 799. Softmax gives exactly the same 24%, 67% and 9%, because the gaps are identical: one point up, one point down. It is like a teacher adding 10 points to every student's exam: every gap between students stays exactly the same.

How it works

Softmax does two things:

1. It makes every score positive and stretches the differences. It raises the number e (about 2.718) to the power of each score, which is called exponentiating. Every extra point of score multiplies the result by about 2.7. That is why the leader pulls so far ahead. How far depends on the size of the gaps: scores a tenth of a point apart end up close to an even split, while gaps of several points hand the leader almost everything.
2. It shares out 100%. It divides each result by the total, so the shares add up to one.

Adding the same amount to every score multiplies every result in step 1 by the same factor, and that factor cancels out in step 2. That is the reason only the gaps count.

In the free course, a factory model sorts moulded parts into four outcomes: ok, short shot (not enough material), flash (too much material, squeezed out of the mould) and burn. Four answers, four raw scores, one softmax to turn them into four chances.

The trick every real implementation uses

The "only gaps matter" rule is more than a curiosity: it saves the computer. Compute 800, 801 and 799 the naive way and e to the power of 800 is too big for a standard computer number. The total becomes infinity, infinity divided by infinity is "not a number", and you get three blanks where three chances should be. No error message and no crash, just a silent hole.

The fix is to subtract the biggest score from all of them first. The gaps do not change, so neither does the answer, but now the largest value is e to the power of 0, which is simply 1. Nothing can overflow.

Why it matters

Softmax is what lets a model with more than two possible answers turn arbitrary scores into percentages you can compare and act on. Training stays simple too: each score is nudged in proportion to the gap between the chance the model predicted and the right answer (100% for the correct one, 0% for the rest). The two-answer case from earlier in the course turns out to be a special case of this all along.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.