Residual connections: why modern AI adds instead of overwriting

What a residual connection is

A modern AI model is a long stack of steps called layers. Each word enters the stack as a list of numbers (its vector), and every layer gets a chance to change it. A residual connection is the rule for how that change is applied: the layer's output is added to what came in, instead of replacing it. New value equals old value plus the layer's contribution. The original is never thrown away; each layer only writes its correction on top.

It sounds too small to matter. The course measures it, and it matters a lot.

What the video shows

The video shows each part of a modern AI adding its contribution on top of what it receives, instead of overwriting it. The original stays intact. And in a test model, replacing instead of adding made the results about four times worse.

The experiment behind that number

The course builds a ladder of small language models, adding one piece at a time, and scores each one with perplexity: roughly, how many words the model is still torn between when it guesses the next one. Lower is better.

One early rung gives each word a summary of all the words before it: a plain average. A later rung uses one attention head, a smarter summary that learns which earlier words deserve more weight.

| How the summary is used | Perplexity | |---|---| | Plain average replaces the word | about 248 | | Plain average added to the word | about 60 | | One attention head replaces the word | 55.5 | | One attention head added to the word | 46.1 |

Replacing threw away the most direct clue the model had, the current word itself, and the result was about four times worse. Even the smarter summary does better when it is added. The course's conclusion: attention should bring context in to a word, not decide what the word is.

An everyday example

Imagine a contract passed around an office for review. In the first office, every reviewer retypes the whole document from memory with their changes worked in. By the tenth reviewer, the author's exact wording is gone. In the second office, reviewers can only add tracked changes. The original text is always there, and each person's suggestions sit on top of it.

The second office is a residual connection. Every layer is a reviewer who can suggest changes but can never delete the original.

Why it helps deep models

The video's hook is about stacking dozens of layers, and the plus sign helps in two directions. Going forward, what earlier layers built is always carried along instead of being rewritten at every step. Going backward, when the model learns, a correction signal (the gradient) travels from the output back through every layer. The plus sign gives that signal a shortcut around a layer that would otherwise block it, which Chapter 6 of the course tests directly.

Every transformer block in the course uses two of these plus signs, one around each of its two main parts.

A common misconception

Residual connections are not a universal repair kit. In the Chapter 6 experiment, a network whose forward signal had already died was not saved by adding them. A shortcut for the learning signal only helps when that signal is the problem; if the values flowing forward are already zero, the shortcut just passes the zero along.

Why it matters

Adding instead of replacing is one of the smallest ideas in modern AI, a single plus sign, and every transformer block depends on it. It keeps each word's own meaning safe while layer after layer adds context on top.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.