Activation functions: the small bend that makes deep networks work

What an activation function is

A neural network is built from layers. Each layer takes some numbers, multiplies them by its settings (called weights) and adds a constant. That kind of step is called linear: if you drew it, you would get a straight line or a flat surface, with no bends anywhere.

An activation function is a small, fixed rule applied to every number after one layer and before the next. Its only job is to bend the line. It sounds like a detail. In fact it is the reason stacking layers is worth doing at all.

What the video shows

The video makes one point: stack as many straight-line layers as you like and the result is still one straight line. Put a small bend between the layers and the network can finally learn curved, complex patterns.

An everyday example

Imagine converting a temperature from Celsius to Fahrenheit, and then from Fahrenheit to Kelvin. That is two steps, each one a "multiply, then add" rule. But you could skip the middle and use a single rule that goes straight from Celsius to Kelvin. Two conversions collapse into one.

Linear layers behave exactly like that. Any two of them can be merged into a single layer that does the same job. The course checks this with real numbers: a two-layer calculation and its merged one-layer version give the same answer down to the last digit, because it is the same arithmetic in a different order. Ten layers, or a thousand, still collapse into one.

Now imagine a rule with a corner in it, like a parking fee that is free for the first hour and then charges by the hour. No single "multiply and add" rule can reproduce that corner. Bends like that, repeated across many layers, are what let a network trace shapes a straight line never could.

Why one straight line is not enough

A classic puzzle is XOR: the answer is "yes" when exactly one of two switches is on, and "no" when both or neither are. Put those four cases on paper and no single straight line separates the yes cases from the no cases. Early single-layer models failed on it for exactly that reason, and piling up linear layers does not help, since the pile is still one line. Add an activation function between the layers and the puzzle becomes solvable.

The main choices, and their weak spots

There is no perfect activation function. Each one swaps one problem for another:

• tanh squeezes any number into the range -1 to 1. For inputs far from zero, whether positive or negative, the curve goes almost flat, so the signal the network learns from barely gets through.
• sigmoid squeezes numbers into the range 0 to 1. It flattens at both ends too, and even at its steepest it passes on only a quarter of the signal. It is still used at the very end of a network when the answer must be a probability, but it has mostly vanished from the inside.
• ReLU keeps positive numbers and turns negatives into zero. It never flattens on the positive side, which is why it took over, although units can get stuck at zero.

Picking one means choosing which of those problems you would rather manage.

Why it matters

Without a bend between layers, a "deep" network is a shallow one in disguise: the extra layers add cost and nothing else. Every deep network relies on some version of this small nonlinear (bending) step, and knowing why helps you see that more layers only means more power when something bends in between.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.