Loss vs metric: what an AI trains on is not what it's graded on

What a loss and a metric are

Building a model involves two different scores, and letting one quietly replace the other is a trap.

• The loss is the score the model learns from. During training, the computer keeps adjusting the model to push this number down. It has to be smooth, meaning a tiny change to the model makes a tiny change to the score, so the computer can tell which direction counts as "better". It also pays attention to how sure the model was, not just to its final answer.
• The metric is the score the model is judged on: the number your team, your customer or your users care about. It can be a plain right-or-wrong count, a business rule, or a tally of defective parts that slipped past the model.

What the video shows

The video's point in one line: models learn from a smooth score that shows them which way to improve, but they are graded by a separate score. The two do not always agree, so you should decide on both before any training starts.

An everyday example

Imagine learning basketball free throws. The metric is simple: in or out. But "out" does not tell you what to change. A shot that missed by a finger and a shot that missed by a metre both score zero.

A good coach tells you something richer: a little too long, a lot to the left. That feedback changes gradually as your shot changes, so you can improve one small adjustment at a time. It plays the role of the loss. You practise with the coach's feedback, and you are judged on baskets made.

Why a model cannot just learn from the metric

Many metrics are step functions: they jump from one value to the next with nothing in between, like pass or fail. Nudge the model a little and such a score usually does not move at all, so it gives no hint about which way to go. That is why training relies on a loss instead. For a model that sorts things into categories, the usual choice is cross-entropy (it has its own video), which gets much worse when the model is confident and wrong.

When the two disagree

The free course shows the gap with three factory models that give exactly the same verdict on all 4,000 test parts. Same verdicts means the same accuracy: 98.30% for each. The metric calls it a three-way tie. The loss does not: 0.0564 for the model as trained, against 0.1549 and 0.1563 for a hesitant copy and an overconfident copy. One model is about three times better by the loss, and the metric cannot see it. (Why that one wins is the story of the Calibration video.)

The reverse can happen too. Because the two are different objects, pushing the loss down does not promise that the number you are judged on goes up.

How to use it

• Pick the metric first. Ask what failure actually costs. On a factory line, a count of defective parts that reached customers may matter far more than overall accuracy.
• Pick a loss that trains well. Smooth, and sensitive to how confident the model is.
• Keep them apart. Report the metric, even when the loss is the number that happens to be on your screen.

Why it matters

A team that watches only the loss can celebrate a number no customer ever sees. A team that watches only the metric has nothing smooth to train on. Naming both up front, and never letting one quietly stand in for the other, is one of the cheapest ways not to fool yourself.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.