Entropy: measuring surprise, and the floor no blind guess can beat

What entropy is

Entropy measures how much surprise a stream of outcomes carries, on average. Put more precisely, it is the average number of bits you would need to report each outcome if you used the cleverest code possible. A bit is the answer to a single yes-or-no question. So entropy asks: on average, how many yes-or-no answers does each new result really contain?

What the video shows

The video compares two sources of outcomes. A fair coin carries one bit of surprise per flip: heads and tails are equally likely, so every result is genuine news. A factory where defects are rare carries almost none, because "fine" is nearly always the answer. That amount, the entropy, is the floor that no blind guess can beat.

An everyday example

Imagine checking a lottery ticket every week. "No win" is what you expected, so it tells you almost nothing. A win would be huge news, but it comes so seldom that, on average, the weekly check carries very little information. If you had to guess the result before checking, "no win" would be your best blind guess, and it would almost always be right. Now compare that with calling a coin toss: heads and tails are a genuine 50/50, so every result is real news. The lottery check has low entropy; the coin toss has high entropy.

The numbers from the course

On the course's factory belt, 1.85% of the parts in the test data are defective. The entropy of that stream is 0.1329 bits per part: less than a seventh of the fair coin's 1 bit.

Now compare two lazy models that never look at any individual part:

• The informed guesser always predicts the defect rate it saw during training, 1.69%. Its cross-entropy, the average cost of its predictions measured in bits, is 0.1330. That is a hair above the entropy.
• The shrugger always says 50/50. It pays exactly 1 bit per part.

The informed guesser knows the overall rate and nothing else, and that is enough to land almost exactly on the entropy floor. The shrugger's extra cost above that floor, 0.8671 bits, is the price of believing the wrong probabilities, which the course calls KL divergence. That idea has its own episode.

What "the floor" means

"Blind" is the key word. A guess that only uses the overall rate cannot do better than the entropy on average. The only way below it is to look at each part, its own measurements, and learn something the overall rate does not tell you. That makes entropy a fair yardstick. If a model's score is barely below it, the model has learned very little about individual cases, however impressive its accuracy looks.

One more detail. When the answer for an example is already known for certain, one outcome at 100%, there is no surprise left, so its entropy is zero. That is the situation during training, where every label is known, and it is why lowering cross-entropy and pulling the model toward the truth end up being the same thing.

Why it matters

Entropy gives you a baseline you can compute before training anything: the best a model can do when it knows the overall picture but nothing about individual cases. The same family of ideas comes back later in the course: Chapter 8 builds perplexity on it, a score for how surprised a language model is by real text, and Chapter 11 uses it in a penalty that stops a fine-tuned model (one given extra training for a specific job) from drifting far from where it started.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.