Batch normalization: the AI layer that grades on a curve

What batch normalization is

Inside a neural network, each layer produces a long list of numbers that the next layer works with. As training goes on, those numbers can drift: too big, too small, or off-centre. A normalization layer keeps them at a healthy size by resetting them. It subtracts an average, divides by the spread, and then applies a scale and a shift that the network learns for itself, so it can undo the reset if that turns out to help.

Batch normalization, often shortened to BatchNorm, is the version that takes that average and spread across a batch: the group of examples the network processes together in one training step. Each of those numbers gets its own average, computed over everyone in the group.

What the video shows

The short sums it up in its hook: an AI layer that grades on a curve. Batch normalization keeps the signals inside the network at a steady size by comparing each example with the rest of its group. It works. The price is that each example's result now depends on who else happened to be in that group.

An everyday example

Imagine a teacher who grades on a curve. Your mark is not about your answers alone; it is about your answers compared with the class that sat the exam with you. Hand in the same essay in a stronger class and you get a lower grade.

Now imagine one student taking the exam alone, on a different day. There is no class to compare with. The teacher's workaround is to use the averages remembered from all the earlier exams. That is exactly what BatchNorm does once the model is being used for real.

The hidden costs

Because each example is measured against its batch-mates, BatchNorm brings some baggage:

• Two modes. A trained model often handles one example at a time, with no batch to average over. So during training the layer keeps a running average of what it has seen, and switches to it afterwards. Training and real use therefore behave differently, and forgetting to put the layer into its evaluation mode is among the most frequent bugs in practice.
• Small batches hurt. An average taken over a handful of examples is a shaky estimate, and the layer gets worse as batches shrink.
• Uneven lengths are awkward. When the examples are sentences of different lengths, the average for, say, the 40th word comes only from the sentences that are at least that long, which may be very few.

There is a small upside too. During training, the noise from batch-mates acts as a mild brake on memorizing the training data, what practitioners call a regularizer.

A common misconception

BatchNorm is often explained with a tidy story called internal covariate shift: the idea that each layer struggles because the signals arriving from earlier layers keep changing as those layers learn, and that BatchNorm cures this. Our course notes that this original explanation is disputed. The layer clearly works. Why exactly it works is less settled than the famous story suggests.

Why it matters

Good starting values set the scale of the signals at the first step, but the weights keep moving, and that careful balance does not last on its own. Normalization layers hold the scale steady throughout training. BatchNorm was one answer. Its dependence on the batch is the reason its close cousin, layer normalization, which measures each example on its own, is the one the transformer uses.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.