Scaled attention: the small division that keeps big models learning

What scaled attention is

In attention, each word scores the words it can see by multiplying two lists of numbers item by item and adding up the results. Those scores then go through softmax, the step that turns them into weights adding up to 100%. Scaled attention adds one small step in between: divide every score by the square root of the list length. For 64 numbers, divide by 8; for 1,024, by 32.

What the video shows

The video shows that as AI models get wider, their attention scores get louder. Too loud, and the focus snaps onto one random word and stops learning. Scaling the scores down keeps attention flexible.

Why wider means louder

Each score is a sum of many small products. The more random pieces you add up, the further the total tends to swing from zero. Measured over twenty thousand random pairs in the course, the typical spread of a score is the square root of the list length. About 2 for 4 numbers, 8 for 64, 16 for 256 and 32 for 1,024. Divide by that square root and the spread goes back to about 1, whatever the width.

Why loud scores freeze attention

Softmax is very sensitive to scale. Small gaps between scores give a soft blend across many words. Blow the same gaps up and you get a hard choice: nearly all the weight lands on one word.

In the course's test, a freshly started attention head (one copy of the mechanism) working with lists of 256 numbers and no division put all of a word's attention on exactly one word out of 64, chosen only by the random starting values. With the division, that same row of weights (one word's full set) spread across roughly 35 words.

A head stuck on one word also barely learns. When softmax is that certain, nudging the scores changes almost nothing, so learning has nothing to push on. At width 1,024 without the division, seven rows in ten were frozen like this before training even began. With it, none were.

An everyday example

Imagine a DJ mixing 64 tracks, where each notch up a fader multiplies a track's volume. With the faders close together you hear a blend, and nudging one audibly changes the mix. Now stretch every gap tenfold: a track one notch above the rest now sits ten above. That track, ahead only by chance, drowns out everything, and nudging the other faders changes nothing anyone can hear. Dividing by the square root squeezes the gaps back to a sensible range.

What happens during training

Frozen heads do not recover; they get worse. In the course's run at width 1,024 without the division, the spread of the scores grew from about 21 at the start to over 5,000, and 99.9% of rows ended up with almost all their weight on one word. With the division, it settled around 3.4: still a soft blend that learning can adjust.

An honest caveat

It does not always matter. With lists of 32 numbers, the narrowest tested, it made no difference; the run without it was even very slightly ahead. At 128 it helped by about 2 points of perplexity, a measure of how many words the model is still torn between when guessing the next one (lower is better). From 256 upwards it was worth 17 to 18 points.

The original transformer paper only said it suspected the scores grow large for long lists. The course's tables show where "large" starts.

Why it matters

One division lets attention keep working as models grow wider. Skip it with narrow lists and you may never notice; skip it with wide ones and many heads lock up for good.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.