Local minima: why AI training can settle for second best

What a local minimum is

Training an AI model means searching for the settings that make it least wrong. Gradient descent does that by walking downhill on an error landscape, a surface where the height is how wrong the model is. It always steps the way that goes down fastest, and it stops when the ground is flat, because then there is no downhill left.

A local minimum is a spot where every direction goes uphill, but which is not the lowest point on the whole map. It is the bottom of a valley, not the bottom of the landscape. The deepest point of all is called the global minimum.

What the video shows

The video's point: when the error landscape has several valleys, walking downhill lands in the nearest one, not the deepest. Then it gives a number from our course: a tiny change in the starting point made the final result 56.7% worse. Hence the hook: start one notch to the left and you get a much better model.

One honest detail: "nearest" is a shorthand. What decides the outcome is which valley the starting point drains into. In the course's own example, the start that finds the deep valley is actually closer to the shallow one.

An everyday example

Imagine rain falling on the ridge of a roof. Two drops land a centimetre apart, one on each side of the ridge. One ends up in the front garden, the other in the back garden. Neither drop made a mistake: each simply ran downhill from where it landed. Where they finished was decided by where they started.

How it works

The course uses a simple curve with two valleys of different depths and a small hump between them. Same step size, same forty steps, two starting points:

| Start | Where it ends | Result | |---|---|---| | 0.11 | the shallow valley, near 0.95 | 56.7% worse | | 0.10 | the deep valley, near -1.05 | the best available |

The top of the hump sits at about 0.101, right between the two starts, so it acts like the roof ridge. That tiny gap decides everything. And the algorithm has no way to notice: from inside the shallow valley, every direction looks uphill, so as far as it can tell, the job is done.

When you have to worry about it

Not always. Some problems have just one valley. Fitting a straight line to data by squared error, the classic line of best fit, gives a convex landscape: a single bowl with one bottom, so walking downhill always reaches the best answer.

Neural networks are different. Their landscapes are not a single bowl, so there are many valleys of different depths, and the starting point decides which one you reach. Gradient descent has no built-in cure for this.

The honest version

The two-valley picture is the classic illustration, and the effect is real. But the course is clear that in practice it matters much less than the drawing suggests. A real network has an enormous number of settings, and each one is a direction the landscape can bend in. In that many dimensions, most flat spots turn out to be saddle points, which go up some ways and down others, rather than true traps. Saddle points have their own short. Later, the course measures how often a small network actually gets stuck.

Why it matters

A training run tells you about the valley it found, not about the whole landscape. Two runs that differ only in where they started can finish at different quality levels, and neither one will report that anything went wrong. Knowing this keeps you from reading too much into a single result.

Learn it step by step, with the two-valley curve, in Chapter 3 of our free course AI From Scratch: Downhill: Gradient Descent, and the Two Steps Everyone Skips.