Induction heads: a two-step circuit behind learning from your prompt

What an induction head is

An induction head is part of a small team inside a language model that spots a repeated pattern and continues it. The team is made of two attention heads, the parts of a model that decide which earlier words to look at. Strictly, the name belongs to the second head, the one that completes the pattern; it relies on an earlier helper head that looks at the word just before each word. The rule the pair carries out is simple: if the text earlier contained A followed by B, and A shows up again, predict B.

What the video shows

The short uses a famous name as its example. The text has mentioned "Ada Lovelace" earlier. When "Ada" shows up again, the model predicts "Lovelace". Two attention steps working together make that happen, and the same kind of teamwork helps AI pick up patterns from examples you put in a prompt.

How the two steps work

In plain terms, the division of labour looks like this:

• Step one, in an earlier layer. A head at each word looks at the word just before it. So the spot holding "Lovelace" now also carries a note: "the word before me was Ada".
• Step two, in a later layer. When "Ada" appears again, a second head goes looking for a spot whose note says "the word before me was Ada". It finds "Lovelace" and passes it forward as the prediction.

Neither head can do the job alone. The first only knows about neighbours. The second needs the notes the first one left behind, and a head in the first layer has nothing to read yet except the words themselves. That is why the trick takes two layers.

An everyday example

Imagine hearing a new song. The first time through, you just listen. The second time the chorus starts, you already know the next line after its opening words, because you remember what followed them last time. Nobody taught you the song in advance; you picked it up from what you just heard.

Prompts work the same way. Imagine typing: "cat: gato, dog: perro, cat:". A model can continue with "gato" without being told any rule, because it has just seen what came after "cat".

Why it matters

Picking up a pattern from examples in the prompt, with no retraining, is called in-context learning. It is what lets you show a model a few examples of a format and have it follow along. Interpretability is the field that tries to read what is happening inside a model, and work in that field from Anthropic links induction heads to a large part of that ability. It also shows that they appear at an identifiable moment during pretraining, the long first phase where a model learns from huge amounts of text.

One honest note on the video's headline, which says this circuit lets AI learn from examples in a prompt. The research links induction heads to a large part of that ability, not all of it. And the course does not take a big model apart to find them: it names them and points to the research, because mapping the wiring of a full-size model is a whole area of research in itself.

What the course does show is the same two-step shape in a tiny model of its own, trained on a pronoun puzzle: one head brings a candidate word forward, and a head in a later layer reads the word that decides whether to keep it.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.