Double descent: the error curve that falls, spikes, then falls again

What double descent is

Every AI model has adjustable settings, called parameters, that it tunes while learning. The classic rule said there is a sweet spot. With too few settings, the model is too rigid to capture the pattern. With too many, it memorizes its training examples, random noise included, and gets worse on new data. So past a certain point, more settings should mean worse results.

Double descent is what happens when you keep going far beyond that point. The error on new data goes down, climbs to a sharp peak, and then goes down a second time. Two descents, hence the name.

What the video shows

The video sets the classic fear, that bigger models were supposed to memorize and fail, against the result of the course's test. With 40 training examples, the error peaks when the model has 40 settings, one per example. After that it falls again, and the model with 5,000 settings does best of all.

The numbers behind it

The course grows a model from 2 up to 5,000 settings, always learning from the same 40 examples, and measures the error on data the model never saw (lower is better):

| Settings | Error on new data | |---|---| | 10 | 1.25 | | 20 | 1.16 (best of the small models) | | 30 | 1.53 | | 40 | 5.81 (the peak) | | 60 | 1.11 | | 200 | 0.66 | | 5,000 | 0.57 |

Three things stand out. First, the peak is real: at 40 settings the error is about five times worse than the best small model. Second, from 40 settings upward, every model fits all 40 training examples perfectly, so the training score cannot tell them apart; only new data can. Third, the biggest model, with 125 times more settings than examples, beats every small model in the test.

An everyday example

Imagine drawing a line through 40 dots on a sheet of paper. With a curve that can only bend a little, you cannot touch every dot, but you can follow their general trend. With exactly enough bend to reach every dot, only one line does it, and it has to swing wildly up and down between them. That is the peak. With a pen that can bend almost any way you like, countless lines touch every dot, and you are free to pick a calm one.

Why training ends up with the calm one is a separate idea, called implicit regularization.

A common misconception

"This model has more settings than it has examples, so it must be memorizing" sounds like common sense. The course shows it is not a sound argument for modern models. The rule worked when models were smaller than the peak. Today's large models, including the language models behind chatbots, sit far beyond it, where the rule turns around.

Why it matters

Double descent explains a real puzzle: how models with vastly more settings than the old rules allowed can still work well on data they have never seen. It also tells you where the danger actually is. The worst place to be is right at the peak, where the model has just barely enough room to fit every example and no freedom left to fit them sensibly.

Learn it step by step, with the full table, in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.