Adam: the optimizer that gives every direction its own stride

What Adam is

Training an AI means adjusting millions of internal settings, a little at a time, so that its answers get less wrong. The recipe that decides how big each adjustment should be is called an optimizer. The simplest one, plain gradient descent (often shortened to SGD), uses a single step size for every setting. Adam keeps a short memory for each setting instead, and sizes that setting's step from its own recent history. It is the usual default for training neural networks today.

What the video shows

The video pictures training as a walk down a narrow valley towards its lowest point. With one fixed stride, you zig-zag. Adam gives each direction its own stride, so it reaches the bottom and rarely crashes.

Why a narrow valley is hard: it is steep from side to side but only gently sloped along its length. A stride small enough to be safe on the steep sides is far too short for the gentle floor, so you bounce from wall to wall and barely move forward.

An everyday example

Imagine an old shower with two taps: the hot tap is extremely touchy, and the flow tap barely responds. If you turn both by the same amount every time, you either scald yourself or wait forever for decent pressure. The sensible thing is tiny nudges on the touchy tap and big turns on the lazy one. Adam does that automatically, for millions of settings at once, based on how touchy each one has proved lately.

How it works

For every setting, Adam tracks two running averages:

• the recent correction it has been receiving, which says which way to move (the maths name is the gradient);
• the recent size of those corrections, squared, which says how big and jumpy they have been.

Each step is the first divided by the square root of the second. A setting that keeps getting large, jumpy corrections has its step scaled down. A setting with small, steady corrections is not held back. The course writes Adam by hand in about ten lines and gets the same numbers as the built-in version in PyTorch, a popular AI toolkit, to within the computer's rounding error.

Then it measures why this matters. It builds a valley where one direction is 1,000 times steeper than the other, and lets each method use its best step size from a range of tries, to keep the contest fair:

| In a 1,000-to-1 valley | Plain SGD | Adam | |---|---|---| | Reaches the target? | No, at any step size tried | Yes, exactly | | Step sizes that blew up | 4 of 8 | 0 of 6 |

When one direction is only ten times steeper than the other, every method does fine. The gap only opens when the landscape is lopsided.

A common misconception

Adam is not the default because it finds better answers. On easy, evenly shaped problems, a carefully tuned plain SGD often matches it or even wins. Adam's real advantage is that it forgives you: it is far less sensitive to the step size you picked. Real networks are much more lopsided than 1,000 to 1 across their millions of settings, so that forgiveness is worth a lot.

Why it matters

The step size has to be picked before training starts, and a bad pick can make a run crawl or blow up. A method that behaves well across a wide range of choices lets people spend their time on the actual problem instead. Its close cousin AdamW keeps all of this and changes one detail: how Adam is combined with the gentle pull that keeps settings small.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.