ReLU: the simple on-off rule inside modern neural networks

What ReLU is

ReLU, short for "rectified linear unit", is the activation function that took over modern neural networks. An activation function is the small rule a network applies between its layers so that stacking layers actually adds something. ReLU's rule fits in one line: if the number coming in is positive, pass it on unchanged; if it is negative, output zero.

Written as maths, it is max(0, z): take whichever is bigger, zero or the input.

What the video shows

The video shows ReLU letting positive numbers through untouched and turning negative ones into zero. The key point is the positive side: unlike the older curves it replaced, ReLU never flattens out there, so the signal the network learns from keeps flowing.

An everyday example

Imagine a sales bonus that equals how far you beat your target. Beat it by 5 units and your bonus is 5. Beat it by 50 and it is 50. Miss it by 3 and you get zero: not a penalty, just nothing. That is ReLU. Above zero it is an honest pass-through; below zero it is a flat floor.

Why it replaced the older curves

To learn, a network needs to know how a small nudge to each of its settings would change its error. That information is called the gradient, and it travels backwards through every layer. Each time it crosses an activation function, it gets multiplied by the slope of that function at that point.

• The older sigmoid curve has a slope of at most 0.25. So even in the best case, each crossing keeps only a quarter of the signal, and those quarters multiply layer after layer.
• tanh does better near zero, but for inputs far from zero, positive or negative, it goes almost flat, and a flat curve lets almost nothing through.
• ReLU has a slope of exactly 1 for any positive input. The signal passes at full strength, however large the number is.

That one property is why ReLU replaced the older curves inside networks. Sigmoid survives mainly at the very end, when the output must be a probability.

A switch, not a dimmer

When the course builds its own small learning engine, ReLU acts as a switch for the learning signal. If the unit's input was positive, the signal flows back through untouched. If it was negative, the signal is blocked completely. There is no halfway.

One small detail: at exactly zero, the corner of ReLU's shape has no true slope. Software libraries simply pick 0 at that point.

The catch

The flat floor has a price. A unit whose input is negative for every example never lets any signal through, so it can stop learning for good. That failure has its own name, dying ReLU, and it is the reason variants such as leaky ReLU and GELU exist.

Why it matters

ReLU is a good example of simplicity winning. A rule anyone could apply by hand removed the flattening that starved older curves of learning signal, and that is why it took over inside neural networks. Understanding why it works also tells you exactly where its one big weakness comes from.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.