Learning rate warmup: why AI training starts with tiny steps

What learning rate warmup is

The learning rate is the size of each step an AI takes while it trains: how far it moves its internal settings after each correction. Warmup means not starting at full size. Training begins with a tiny learning rate and raises it gradually over the first few hundred steps, until it reaches its normal value.

What the video shows

At the very start of training, an AI's estimates are still rough. Taking full-size steps on them can wreck the run. The video shows warmup starting with tiny steps and growing them gradually, which is why training begins in slow motion.

Why the start is fragile

Adam, the default recipe for training neural networks, sizes each setting's step using a running average of how large its recent corrections have been. That average needs history. On the first step it rests on a single correction, so Adam's estimate is closer to a guess than a measurement. A full-size step based on a bad guess can knock the model's settings far away from where they began.

And where they began matters. Before training, a model's settings are given carefully chosen starting values, and in a deep network those starting values are often what makes learning possible at all. One wild early step can throw that good start away before the real learning has even begun.

An everyday example

Imagine a fitness app that sets your training plan from the average of your recent runs. On day one it has seen exactly one run, maybe on a day you were ill or unusually fresh. If it built a hard plan straight from that single run, it could push you far too hard. A sensible app starts gently and ramps up as it learns your real pace. Warmup gives the optimizer, the recipe that turns corrections into steps, the same grace period before it takes big strides.

When you would use it

Warmup is a short, cheap addition to a training schedule, and it is most useful with optimizers like Adam that rely on those running estimates. It is usually paired with cosine decay, which shrinks the step size toward zero at the other end of the run: start small, grow, then settle.

A common misconception

Warmup is not a law of nature. Later in the course, in Chapter 9, it turns out to be what lets the original design of the transformer (the architecture behind modern chatbots) train at all. That design places a steadying layer, called LayerNorm, after each of its building blocks. Moving that layer to sit before each block instead, a variant known as pre-norm, removes the need for warmup. In that case, warmup was compensating for a weakness in the design, and a better design made it unnecessary.

Why it matters

The first few hundred steps are a tiny fraction of a training run, but they are where it is easiest to break something. A slow start costs almost nothing, while a wrecked start can cost the whole run. Knowing why warmup helps also tells you when to question it: if the start is fragile because of the design, as in the Chapter 9 case, changing the design can remove the need for it.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.