Multi-head attention: why AI reads with several spotlights at once

What multi-head attention is

In attention, each word decides how much to take from every earlier word. One attention head makes that decision once: for each word it produces a single set of weights, a single spotlight. Multi-head attention runs several heads side by side, each with its own spotlight, and then combines what they found.

What the video shows

The video starts from a limit: one attention spotlight can focus on one thing, but a word usually needs several clues. Multi-head attention runs many spotlights in parallel. The video sums it up with a comparison in which four spotlights beat one, 75% to 35%. That figure does not appear in the course chapter, so read it as an illustration of the idea rather than a measured result. The chapter's own measurement, on a small language model, is below.

Why one spotlight is not enough

Imagine the model has read "the animal that crossed the wet street" and must guess what comes next. It helps to track several things at once: who the sentence is about (the animal), what kind of word fits the grammar at this point, and what the very last word was.

One head gives each word one set of weights that add up to 100%. It can spread them thinly or pile them on one word, but it cannot put a strong focus on three places at once. Several heads can: each one is free to follow a different clue.

An everyday example

Think of a football broadcast. One camera follows the ball, another stays on the goalkeeper, a third watches the coach on the sideline. Each camera sees less than a single wide shot would, but each can stay focused on its own subject. A director in the control room mixes the feeds into the one picture you watch.

Multi-head attention works the same way: several narrower spotlights, each following its own clue, and one final mixing step.

How it works

• Split, do not add. Each word is described by a list of numbers, and the length of that list is the model's width. The heads share that width instead of each getting a full copy. In the course's small model, a width of 128 is split between 4 heads.
• Each head matches on its own. Every head does its own query, key and value matching, so each is free to look for something different.
• Put them back together. The heads' outputs are placed side by side and mixed by one more learned grid of numbers, which returns a single result per word.

What the course measured

The course scores models with perplexity: roughly, how many words the model is still torn between when it guesses the next one. Lower is better. Parameters are the adjustable numbers a model learns.

| Model | Parameters | Perplexity | |---|---|---| | One attention head | 328,704 | 46.1 | | Four heads instead of one | 345,216 | 43.2 |

Four heads improved the score by about 3 points for roughly 16,500 extra parameters, about 5% more. The course's table of larger models pairs a width of 768 with 12 heads and a width of 4,096 with 32.

More heads, almost the same size

Adding heads does not grow the model in proportion. Because the heads split the width rather than adding to it, going from one head to four barely changed the size. What changes is how many different things the model can pay attention to at once.

Why it matters

Multi-head attention is the version used inside transformers. It lets one layer follow several clues at once, such as the subject, the grammar and the previous word, instead of forcing a single compromise. Attention itself only moves information between words; other parts of each block work on each word on its own.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.