Dying ReLU: when parts of a neural network quietly switch off

What dying ReLU is

ReLU is the activation function that took over inside neural networks (the small rule applied between layers): positive numbers pass through, negative numbers become zero. That simple rule has one serious side effect.

A network is made of many small calculators called units (or neurons). A unit can end up receiving a negative input for every single example in the training data. From then on it outputs zero, always. And because of the way networks learn, it never gets a signal telling it how to recover. The unit is dead, and it stays dead.

What the video shows

The video explains what happens to a unit like that. It only ever receives negative inputs, so it always outputs zero, gets no signal to recover, and stays dead for good. It also points to the fix: versions of ReLU that leak a little on the negative side keep the unit alive.

Why a dead unit cannot come back

A network learns by sending a correction signal backwards from its error to every setting. With ReLU, that signal has a simple rule at each unit: if the unit's input was positive, the correction flows through; if it was negative, it is blocked completely, because the slope of ReLU's flat part is zero.

Now suppose a unit's settings get pushed to a place where its input is below zero for every example. Every correction arrives, hits that flat floor and stops. The settings feeding the unit never change again, so its input stays negative, so it keeps outputting zero. It is a closed loop with no exit.

An everyday example

Imagine an employee who only receives feedback on the reports they hand in. One month they stop handing anything in. Now nobody sends feedback, because there is nothing to comment on, and without feedback nothing tells them how to start again. They are still on the payroll, but they contribute nothing, and no alarm ever goes off.

Why nobody notices

A dead unit does not crash anything. The course describes this happening quietly in the middle of training: the network keeps running, the numbers keep updating, and part of its capacity has simply gone. The model is effectively smaller than the one you set out to train, and nothing on screen tells you.

How to spot it and fix it

The definition gives you the test. Run the whole training set through the network and, for each unit, check whether it ever produced anything other than zero. A unit that was zero for every example is dead.

The usual fix is to change the rule so the negative side is not completely flat:

• Leaky ReLU lets a small trickle through for negative inputs instead of a hard zero, so a correction signal can always find its way back.
• GELU is another variant that the course lists as a patch for the same problem.

Why it matters

Dying ReLU is a reminder that every activation function trades one weakness for another. ReLU fixed the fading signal of the older, smoother curves, but in exchange it can switch units off permanently. Knowing this failure exists is the first line of defence: a simple check can reveal a network that is quietly running on only part of its engine.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.