Cross-entropy: the score that makes AI pay for confident mistakes

What cross-entropy is

When a model learns, it needs a single number that says how badly it is doing. That number is called the loss, and training means pushing it down. Cross-entropy is the loss used for models that answer yes-or-no questions with a probability.

It does not just ask "were you right?". It asks "how much probability did you give to what actually happened?" Give the real answer a high probability and the cost is tiny. Give it a low one and the cost explodes.

What the video shows

The video shows that these models are scored on how sure they were, not just on whether they were right. Being confidently wrong costs 44 times more than being right and fairly sure.

Here is the price list behind that claim, for one example where the real answer is "yes":

| What the model said | Cost | |---|---| | 90% yes: right, and fairly sure | 0.105 | | 50/50: refuses to commit | 0.693 | | 1% yes: wrong, and 99% sure of it | 4.6 |

The middle row is the fixed fee for sitting on the fence. And the last row has no ceiling: the more certain the model is about a mistake, the higher the bill climbs.

An everyday example

Imagine a quiz where you must bet on every answer. Say "I'm fairly sure it's Paris" and be right, and you lose almost nothing. Say "no idea, it's a coin toss" and you pay a moderate fee. Say "I'm 99% sure it's Rome" when the answer was Paris, and you lose a fortune. Play a few rounds under those rules and you stop bluffing. You only sound certain when you have a good reason to be.

How it works

For each example, the cost is minus the logarithm of the probability the model gave to the right answer. You do not need to calculate a logarithm to follow the idea. All you need is its shape: it barely moves when the probability is close to 1, and it dives without limit as the probability gets close to 0. So:

• a probability near 1 for the truth costs almost nothing;
• a probability near 0 for the truth costs a lot, and there is no maximum.

This formula was not picked because it is handy. The course starts from the only sensible description of a yes-or-no label: a weighted coin flip, where there is some chance of "yes". Ask how likely the real labels were under the model's chances, take the logarithm, flip the sign, and cross-entropy is what you get. There was no other candidate.

There is a second way to read the same idea, from information theory: cross-entropy counts the bits, or yes-or-no answers, you spend when you describe data using a code designed for different probabilities than the real ones. The episodes on entropy and KL divergence cover that view.

A common misconception

A loss is not the same as a grade. Two models can both be right on every example and still get different cross-entropy scores, because one was hesitant and the other was sure. Cross-entropy rewards being right and being honest about how sure you are.

Why it matters

Under cross-entropy, confident mistakes dominate the bill, so the fastest way for a model to improve is to stop making them. That pushes the model away from bluffing. It also explains why training is so sensitive to a few badly wrong examples: each one can cost more than dozens of correct ones. Why not simply use squared error instead? That comparison has its own episode, and the answer is about speed.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.