Cross-entropy vs MSE: why squared error stalls on its worst mistakes

What the comparison is about

To train a model you need a loss: a number that says how wrong the model is, which training then pushes down. For models that answer yes-or-no questions with a probability, there are two obvious candidates.

• Squared error (MSE, short for mean squared error) takes the gap between the prediction and the truth and squares it.
• Cross-entropy charges minus the logarithm of the probability the model gave to the right answer.

Both can train a model. The real question is how fast, and at which moments.

What the video shows

The video shows that a classifier scored with squared error barely reacts when it is badly wrong, while cross-entropy reacts hardest at exactly that moment. In a race between the two, one finished in 50 rounds of training and the other needed 500.

An everyday example

Imagine a heating thermostat that works hard when the room is a couple of degrees too cold, but barely switches the heating on when the room is freezing. It is not broken, exactly: leave it long enough and the room does warm up. It just does the least when the room needs it most. That is squared error sitting on top of the S-shaped curve that turns a score into a probability.

How it works

Training adjusts the model using the gradient: a signal that says which way to change each setting, and how hard. A strong signal means fast learning. A weak one means slow learning.

With squared error, that signal carries an extra factor, p × (1 − p), where p is the model's probability. That factor is close to zero whenever p is close to 0 or 1, in other words, whenever the model is confident. Confident and right, that is harmless. Confident and wrong, it is a trap: the model gets almost no signal to fix itself.

With cross-entropy, that factor cancels out. The signal is simply prediction minus truth, so the worse the mistake, the stronger the push, up to a maximum size of 1.

Here is how much weaker the squared-error signal is, for an example whose true answer is "yes":

| Model's raw score | Probability it gives to "yes" | Squared error's signal is weaker by | |---|---|---| | 0 | 0.5 | 2 times | | -2 | 0.12 | 4.8 times | | -4 | 0.018 | 28 times | | -8 | 0.0003 | 1,491 times |

The race

The course runs both losses from an identical start: 2,000 examples split evenly between the two answers, settings chosen to be confidently wrong, and the same learning speed. Only the loss differs. Each round is one full pass through the training data.

• Cross-entropy is finished by round 50, at about 78% accuracy.
• Squared error is still at 24% accuracy after 100 rounds. On evenly split data that is worse than a coin toss. It finally escapes at around round 500 and ends up in the same place.

A common misconception

Squared error is not wrong for this job. It gets there in the end. The textbook argument against it (it assumes the wrong kind of noise in the labels) is correct, but it rarely changes anyone's mind. The argument that does is speed: squared error is slowest exactly when the model most needs to move. On this tiny model that costs 450 extra rounds. On a network with a hundred layers, where some part is always confidently wrong, the course warns it can cost you the whole training run.

Why it matters

The choice of loss is not a detail you can leave to defaults. It decides how much the model learns from its worst mistakes, and whether it learns fast enough to be useful.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.