Bias vs variance: why a model that fits every point can still fail

What bias and variance are

Every prediction model makes mistakes. The classic way to understand them is to split a model's expected error into three parts:

• Bias: error from a model that is too rigid. It is too simple to represent the real pattern, so it misses on the examples it trained on and on new ones alike. This is called underfitting.
• Variance: error from a model that is too flexible. It bends to fit the random noise of the particular examples it was trained on, so its answers depend heavily on which examples it happened to see. This is called overfitting.
• Noise: randomness in the data itself. No model can remove it.

Simple models tend to suffer from bias, very flexible ones from variance, and the classic advice is to aim for the sweet spot between them.

What the video shows

The video contrasts the two failures: too simple and a model misses the pattern, too flexible and it chases noise. Its punchline comes from an experiment in our course, where a curve that passed through all 12 training points guessed 61 at a spot where the real answer was about 0.

The experiment behind it

Take twelve points from a smooth curve and add a little random noise. Now fit curves of growing flexibility to them. A curve's degree measures how much it can bend: degree 1 is a straight line, and each step up allows more wiggles.

• Degree 1 cannot follow the shape at all. It is bad on the training points and bad on new ones.
• Degree 5 does best on new points it has never seen.
• Degree 11 goes exactly through all twelve training points, with an error of zero on them. On new data it is eight times worse than degree 5. Asked for a value just past the edge of the data, it answers +61. The true value there is about 0.

The degree 11 curve never learned the shape. It learned those twelve dots, and everywhere else its answers are whatever the maths happens to produce.

An everyday example

Imagine buying clothes. A one-size-fits-all T-shirt is high bias: it ignores your actual shape, so it never fits anyone particularly well. Now imagine a tailor who measures you once, while you slouch with a phone in your pocket, and cuts a suit that follows every detail, bulge included. It fits that moment perfectly. The next day, standing straight with an empty pocket, it fits badly. That is high variance: the suit learned the accidents of one fitting instead of your body.

How to tell which one you have

• Too simple (high bias): the model does badly on its training data and on new data alike.
• Too flexible (high variance): the model looks excellent on its training data and much worse on new data. A near-perfect training score next to a weak score on unseen data is the warning sign.

The usual cures for variance, such as penalties on large settings, dropout, early stopping and making extra training examples, are separate concepts in this series.

Why it matters

A perfect score on examples the model has already seen proves very little. What counts is how it does on data it has never met. Bias and variance give you names for the two ways to fail, and a direction for fixing each one: more flexibility for the first, more restraint or more data for the second.

One honest note: our course presents this as the classic story and, right after it, shows that the story is incomplete. That twist, known as double descent, is a separate concept in this series.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.