KL divergence: the price of believing the wrong odds

What KL divergence is

Every model that predicts chances holds a belief about how likely each outcome is. Reality has its own odds. When the two differ, the model wastes information, and KL divergence is the size of that waste. It is counted in bits, where one bit is the information in a single yes-or-no answer when both answers are equally likely.

A KL divergence of zero means the model's odds match reality exactly. The bigger the number, the more the model pays, on average for each prediction, for being wrong about how the world works.

What the video shows

The video makes one idea visible: believing the wrong odds has a price, and that price can be measured. When a model's beliefs drift away from reality, it wastes information. KL divergence puts a number on that waste, and training a model works by shrinking it.

An everyday example

Imagine a friend picks one snack from a jar every day, and you have to find out which one by asking yes-or-no questions. The jar is mostly cookies. If you know that, your first question is "Is it a cookie?", and most days you are done after one question.

Now imagine you wrongly believe every snack in the jar is equally likely. You ask your questions in a worse order and, over many days, you need more of them on average. Count each question as one bit. The extra questions you ask on average, purely because your odds were wrong, are roughly the KL divergence.

How it works, with real numbers

In the free course, a model inspects parts on a factory belt, and 1.85% of the test parts are defective. Three numbers tell the whole story:

• The floor: 0.1329 bits per part. This is the entropy (it has its own video in this series): the unavoidable cost of not knowing which individual part is bad, even if you know the true odds perfectly.
• The full bill of a lazy model: exactly 1 bit per part. A model that always says 50/50 treats a rare defect like a coin toss.
• The gap: 0.8671 bits. That is the KL divergence, the pure cost of believing the wrong odds.

Add the floor and the waste and you get the bill: 0.1329 + 0.8671 = 1 bit. That total is the cross-entropy, what a model actually pays when it uses its beliefs on real data. For contrast, a model that simply reports the defect rate it saw during training, 1.69%, pays 0.1330 bits. It sits almost exactly on the floor, so its waste is close to zero.

Why training shrinks it

During training, every example comes with one known answer: this part is defective, that one is fine. A known answer carries no uncertainty, so its entropy is zero. With no unavoidable part left, the loss the model is trained on (the cross-entropy) and the KL divergence become the same number. Lowering the loss is literally pulling the model's odds toward the truth.

Why it matters

KL divergence does not stay in this chapter. Later in the course it returns as a penalty used when a finished model is adjusted further: it keeps the new version from drifting too far from the original. It is also friendlier than its reputation. Once you see it as wasted yes-or-no questions, the formula is just a bill you can add up by hand.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.