Autodiff: how a program works out its own slopes, exactly

What autodiff is

Training a neural network means nudging millions of internal settings so that its mistakes shrink. To know which way to nudge each one, you need its slope: how much the final error moves when that single setting moves a tiny bit. (The course calls this the gradient.)

Automatic differentiation, or autodiff, is the trick that produces all those slopes without anyone doing calculus by hand. Every number in the calculation keeps a note of which numbers produced it, and every operation (add, multiply and so on) knows one small rule about how a change passes through it. The computer chains those small rules together, and the slopes come out exact, not estimated.

What the video shows

The video contrasts two ways of working. The old way is to derive every slope by hand, formula after formula. The autodiff way is to let each number remember its origin and let each operation carry its own rule. Once that is in place, the slopes compute themselves, and they match what industrial tools produce.

An everyday example

Imagine a spreadsheet for a party budget. The total is the number of guests times the price per head, plus a fixed room fee. You want to know: if I add one guest, how much does the total go up?

You never need a formula for the whole sheet. Each step only answers for itself:

• The plus step says: a change that reaches me passes straight on to both of my inputs. One more dollar of room fee is one more dollar of total.
• The times step says: a change in one of my inputs gets scaled by the other input. One more guest adds exactly the price per head.

Chain those two local answers and you have the slope. Autodiff does exactly this, automatically, for calculations with millions of steps.

How it works

The course builds a working autodiff engine in roughly 120 lines of Python. Each operation gets its own rule, and side by side they form a neat pattern:

| Operation | What it does to the slope signal | |---|---| | Addition | copies it to every input | | Multiplication | scales each input's share by the other input's value | | ReLU (a switch that turns negatives into zero) | lets it through or blocks it | | tanh (an S-shaped squashing curve) | shrinks it, or at most leaves it the same size |

Two details keep the answers right. First, when one number feeds into several later steps, the slopes arriving from all those routes are added up. Second, the engine visits the steps in reverse, from the final error back to the start, so each slope is finished before it is passed further back. Visit them in the wrong order and you get a wrong answer with no warning at all.

Why backwards? Working forwards tells you how one input affects everything, so you would need a full pass for every single setting. Working backwards tells you how everything affects the one final error, in a single pass. A network has millions of settings and one error, so backwards wins by a landslide. The price is memory: every in-between value has to be stored until the backward pass comes back for it.

Why it matters

Autodiff is what turns training a huge network from a calculus nightmare into a routine job. You write the forward calculation, and the slopes come along automatically: you pay in memory, not in hand-written formulas. The idea is also older than modern AI: Linnainmaa published the reverse technique in 1970, sixteen years before the 1986 Nature paper that made it famous among neural-network researchers.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.