Calibration: when an AI's confidence matches how often it is right

What calibration is

A model rarely just says "yes" or "no". It usually says how sure it is: 60%, 95%, 99.9%. A model is calibrated when you can take those numbers at face value: when it says 80%, it turns out right about 80% of the time. In short, its confidence matches how often it is actually right.

Being right and being honest about how sure you are are two different skills. A model can get the same answers right as another one and still be far less trustworthy, because its percentages lie.

What the video shows

Three models give exactly the same answers, so they have exactly the same accuracy. Only one of them has confidence that matches reality. Scored with a measure that looks at confidence, not only at the final answer, that one comes out about three times better than the other two: its penalty is roughly a third of theirs.

An everyday example

Imagine three weather forecasters who make the same call every day, rain or no rain, and get the same number of days right. The first always hedges: "maybe, 55%". The second always says "100% certain". The third says 90% on some days and 60% on others.

The first is useless for planning. The second sends you out without an umbrella, fully confident, on the day it gets wrong. The third is the one you want, as long as its 90% calls really come true about nine times in ten, and its 60% calls about six times in ten. That forecaster is calibrated.

How the course builds the three models

The free course takes a trained model that sorts factory parts into good and defective, then makes two copies by scaling its raw scores. Multiplying every score by 0.35 gives a hesitant copy; multiplying by 4 gives an overconfident one. The verdict (good or defective) depends only on whether a score is above or below zero, and scaling by a positive number never flips a score from positive to negative, so all three give the same verdict on every one of the 4,000 test parts. Accuracy is 98.30% for all three.

The cross-entropy score, which reads how sure the model was and punishes being sure and wrong, tells them apart:

| Model | Accuracy | Cross-entropy (lower is better) | Penalty on its worst part | |---|---|---|---| | Hesitant | 98.30% | 0.1549 | 2.80 | | As trained | 98.30% | 0.0564 | 7.82 | | Overconfident | 98.30% | 0.1563 | 27.63 |

The hesitant copy pays a small fee on every part, including the thousands it gets right. The overconfident copy pays almost nothing when right and a fortune when wrong: a single part costs it 27.63 units on its own. They end up with nearly the same total from opposite directions. The model as trained, whose probabilities are calibrated to the data, lands at roughly a third of either.

How to spot a badly calibrated model

• Do not trust accuracy to tell you. It only checks the final answer, so it cannot see the difference at all.
• Use a score that reads confidence. Cross-entropy does, which is how the course separates the three models.
• Check the percentages against reality. Gather the cases where the model said about 80% and count how often it was right. Far fewer than 80% means overconfidence; far more means it is too timid.

Why it matters

As soon as someone acts on a model's percentage, and not only on its answer, calibration is what makes that percentage mean something. An overconfident model is at its most dangerous exactly where it is wrong. (Why the score a model trains on differs from the one it is judged on is a separate idea, covered in the Loss vs Metric video.)

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.