Cosine decay: how AI training slows down to land on the answer

What cosine decay is

Training an AI model is a long series of small corrections. After each batch of examples, the model nudges its internal settings (the numbers it uses to make predictions) in whatever direction lowers its error. How big each nudge is depends on one number, the learning rate. In plain words, it is the step size.

Cosine decay is a plan for how that step size changes over time. It starts with big steps and lowers them smoothly toward zero, following the shape of a cosine curve, so the last steps of training are tiny. It normally comes right after a short warmup, a brief phase at the very start where the steps grow from almost nothing (warmup is its own concept in this series).

What the video shows

Big steps are great at the beginning: they bring the model close to a good answer quickly. But once it is close, the same big steps become the problem. Each one overshoots, so the model keeps jumping from one side of the answer to the other. The video shows cosine decay shrinking the steps gradually, so the model settles in instead of bouncing, with the shrinking timed to finish exactly when training finishes.

An everyday example

Imagine parking a car. On the main road you drive at normal speed, because you are still far away. As the parking space gets closer you slow down, and for the last metre you barely move. If you kept road speed the whole way, you would sail past the space, reverse, overshoot again, and never park.

Training is the same. Picture the model's error as a valley, with the best settings at the lowest point. High up the slope, long strides are efficient. Near the bottom, a long stride carries you straight up the other side.

How it works

The step size follows half of a cosine wave, which has a gentle shape:

• At the start it stays close to full size for a while, so early progress is fast.
• In the middle it falls more quickly.
• Near the end it glides softly down to almost zero, so the final corrections are small and precise.

There are no sudden jumps and no manual switching between speeds. You choose the starting step size and the total length of training, and the curve handles the rest.

A common mistake: the wrong length

The whole schedule is built around how long you plan to train. Our course comes back to this in Chapter 10: if you set up a schedule for a longer run, with more data, and then stop early, you stop while the step size is still far too high. The model never reaches the gentle final phase, so it is still bouncing when you take it. The curve only helps if its end matches the real end of training.

Why it matters

If the last steps of a run are as big as the first ones, the model spends its final stretch hopping around the best answer instead of resting on it. Cosine decay is a simple fix: fast progress early, a careful landing late. That is why our course presents it right alongside warmup and gradient clipping (which scales down any correction signal that grows too large), as small additions to Adam, the default method for applying the corrections.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.