Dropout: why switching off random parts of an AI helps with new data

What dropout is

A neural network is built from many small units, sometimes called artificial neurons. Each one takes in numbers, does a simple calculation and passes a signal on. Dropout is a training habit: at every training step, a random selection of those signals is switched off (set to zero), as if those units were not there. At the next step, a different random selection goes dark.

Because the missing units keep changing, no unit can count on any particular partner being present. Each one has to be useful on its own.

What the video shows

The video's idea is simple: switch random parts of the AI off while it trains, and no part can rely on another. It compares this to a sports team practising with players missing, where everyone learns to cover. The reason is easy to see: if any player might be absent on a given day, the team cannot build its game around one star.

A note on the word "smarter" in the video's hook: here it means better on new data the model has never seen, not better at the examples it practised on.

An everyday example

Imagine a small restaurant kitchen where only the head chef knows how to make the sauce. The day the chef is ill, service falls apart. Now imagine a manager who sends a different cook home at random on each practice day. Soon several people can make the sauce, the grill and the dessert, and the kitchen copes with whoever turns up.

A network trained without dropout can end up like that first kitchen: groups of units that only work in one exact combination.

The problem it fights: overfitting

Overfitting means a model memorises its training examples, random quirks included, instead of learning the general pattern. It then looks excellent on the data it practised on and disappoints on anything new.

Dropout makes tight, fixed partnerships between units impossible to rely on, because any partner might vanish at the next step. Our course lists it among the standard tools against overfitting, next to penalties on large weights, early stopping and data augmentation (making extra training examples from the ones you already have).

When it is switched off

Dropout is not always used. Our course points out, in Chapter 9, that it is often set to zero in large pretraining runs: the huge first phase of training, where a model reads enormous amounts of text. There, the model sees each token (a small piece of text) only once, so it has little chance to memorise anything, and there is little need for the extra protection. The same chapter shows exactly where dropout goes inside a transformer, the type of network used by modern language models.

Why it matters

Dropout is a simple idea with a practical payoff: a network in which no part can depend on one fixed partner has a harder time memorising its practice examples. It is also a good lesson in reading training recipes. Every technique solves a specific problem, and it makes sense to switch it off when that problem is absent.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.