Saddle points: where AI training looks stuck but has a way down

What a saddle point is

Training an AI model means walking downhill on an error landscape, where the height is how wrong the model is. The walker, an algorithm called gradient descent, only feels the slope under its feet and keeps stepping downhill. When the ground is flat, it stops.

Flat ground usually suggests the bottom of a valley. But there are three kinds of flat spot:

• a valley bottom, where every direction goes up;
• a hilltop, where every direction goes down;
• a saddle point, where some directions go up and others go down.

The name comes from a horse saddle. Sit in the middle and it rises towards the front and the back, but falls away to the left and the right.

What the video shows

The video makes one clear point: some flat spots are not valleys at all. The ground rises one way and falls another, like a saddle. And in huge networks, most flat spots are like this, so there is usually a way down. That is what the hook means by "looks stuck, but isn't": a saddle is not a dead end. It does not mean the way out is easy to see, as the next sections explain.

An everyday example

Think of a mountain pass. If you drive over it, the pass is the highest point of your road. If you hike along the ridge from one peak to the next, the very same spot is the lowest point of your route. Standing there, the ground is flat, yet it is a top and a bottom at once, depending on which way you face. That is a saddle.

How it works

The textbook saddle is a surface whose height is one coordinate squared minus the other coordinate squared. Right at the centre the slope is zero. Along one axis, the centre is a low point. Along the other axis, it is a high point.

Here is the catch. Gradient descent only reads the slope, and at all three kinds of flat spot the slope is exactly zero. So it cannot tell a saddle from a valley bottom or from a hilltop. A saddle does have a way down, but a walker that only reads the slope gets no hint of which way it is.

Why big networks are full of them

Our course cites research by Dauphin and colleagues: in the very high dimensions of real networks, most flat spots turn out to be saddles rather than traps.

A rough way to picture why: every setting of the model is one more direction you could walk in, and real networks have an enormous number of settings. To be a true valley bottom, the ground has to rise in every single one of those directions at once. The more directions there are, the harder that becomes, and the more likely it is that at least one of them still leads down. It is an intuition, not a proof.

A common misconception

The popular worry is that training gets trapped in a poor valley, a local minimum, which has its own short. That picture comes from drawings with only two or three dimensions. The course's view is that in real networks those traps matter much less than the drawings suggest, precisely because most flat spots are saddles.

Why it matters

A flat spot with a way down is not a dead end. Knowing that most flat spots in big networks are saddles makes training huge models less mysterious: the landscape holds fewer traps than simple pictures imply. It also explains why flat ground on its own proves nothing: the slope is zero at a valley bottom, at a hilltop and at every saddle alike.

Learn it step by step in Chapter 3 of our free course AI From Scratch: Downhill: Gradient Descent, and the Two Steps Everyone Skips.