Computational graphs: how AI tracks where every number came from

What a computational graph is

A computational graph is a calculation drawn as a map. Each value is a dot, and arrows show which values were combined, and how, to produce it. The key idea is memory: every number remembers its parents (the numbers that produced it) and the operation that joined them.

That sounds like bookkeeping, and it is. But it is the bookkeeping that makes training neural networks possible, because it lets a program work out how much each input influenced the final result.

What the video shows

The video's point is that any calculation can be drawn as a network of small steps. To find how much an input affects the result, you multiply along each path from the input to the result, and then add up all the paths.

An everyday example

Imagine planning a party. Food costs 12 per guest, drinks cost 5 per guest, and the total is food plus drinks. As a graph, "guests" feeds two boxes, "food" and "drinks", and both of those feed "total".

How much does one extra guest change the total? Follow each path:

• through food: 1 extra guest × 12 = 12
• through drinks: 1 extra guest × 5 = 5
• add the two paths: 12 + 5 = 17

Multiply along a path, add across paths. That is the whole rule the video shows.

How it works

The clever part is that every step only needs local information. An addition box simply hands the influence it receives to each of its inputs, unchanged. A multiplication box scales each input's share by the value of the other input: that is why the guests got 12 through the food box. No box needs to know anything about the rest of the map.

In the course's own small engine, each number carries four things: its value, how much it influences the final result (called its gradient), the numbers it came from, and a tiny rule for passing influence back to them. Influence is always added on top, never overwritten, so a value that feeds two places collects both contributions by itself.

Why the order matters

Now imagine the guest count itself came from invitations, with 80% of people saying yes. Each extra invitation is worth 17 × 0.8 = 13.6 on the total. But if you passed the influence back to "invitations" before the drinks path had reported in, you would get 12 × 0.8 = 9.6: a wrong answer, and no error message.

So the program first sorts the map so that each value is handled only after everything that depends on it has reported back. Walk it in the wrong order and the answers are silently wrong.

Why it matters

A neural network is one enormous computational graph, with millions of settings feeding a single error score. The map is what lets training ask every setting, in one backwards walk, how much it contributed. That walk has its own name, backpropagation. The price is memory: every in-between value must be kept until the backwards walk reaches it, which is why those in-between values take up most of the memory during training.

Learn it step by step in Chapter 5 of our free course AI From Scratch: Backpropagation From Scratch: The Engine, Then the Network.