Weight initialization: the starting numbers that decide if AI learns

What weight initialization is

Before a neural network learns anything, its internal settings, called weights, have to start somewhere. They are filled with random numbers. Weight initialization is the decision about how big those random starting numbers should be.

It sounds minor, but the starting size decides whether a signal can travel through the network at all, and so whether training ever gets going.

What the video shows

The short makes the case in about 25 seconds. The settings start random. If they start too small, the signal fades away as it passes through the layers. If they start too big, it blows up. With the right starting size, the same network goes from 50% accuracy, no better than a coin flip, to 100%.

An everyday example

Imagine photocopying a photocopy, six times in a row. If the machine is set too light, each copy comes out much fainter than the one before, and by the fourth copy the page is blank. Set too dark, each copy gains ink until the page is a black smear. Only a setting that keeps each copy as dark as the last lets the picture survive.

A network's layers behave like that chain of copies. Each layer hands its output to the next, so whatever the starting size does to the signal, shrinking it or swelling it, happens again at every layer.

What goes wrong at each extreme

Our free course tests this on a network with six layers, each taking 128 inputs. "Typical size" below means the standard deviation, a measure of how spread out the random numbers are.

• Too small (typical size 0.01). This feels like the cautious choice, and it is fatal. The signal shrinks at every layer, and by the fourth layer it has faded to zero. From then on every input gets the same output. An output that never changes sends back no useful correction, so the network stays stuck at coin-flip level however long it trains.
• Too big (typical size 1). The units are pushed against the edge of their range, and the corrections sent backwards during learning grow out of control. That failure has its own name: exploding gradients.

The rule behind the right size

Each unit in a network adds up its inputs, each one multiplied by a weight. The more random terms you add together, the more widely the total swings. So the more inputs a unit has, the smaller each weight should be. The rule that keeps the signal at the same size from one layer to the next: make the variance of the weights (the square of their typical size) equal to 1 divided by the number of inputs.

Two well-known recipes build on that rule:

• Xavier initialization balances what the forward pass needs (signals going in) with what the backward pass needs (corrections coming back). Its variance is 2 divided by the sum of the layer's inputs and outputs.
• He initialization is for ReLU units, which switch off about half of what they receive and so halve the variance. To make up for it, it uses 2 divided by the number of inputs.

For 128 inputs, Xavier gives a typical size of about 0.088. The stuck network used 0.01. The gap is roughly a factor of nine, and it is the entire difference between a network that learns and a network that never does.

Why it matters

Because it happens before training starts, initialization is easy to overlook when a model refuses to learn. The course's advice is to look before guessing: measure how big the signal is at each layer. A signal that shrinks to nothing, or one that swells, points straight at the starting values.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.