RoPE: how spinning arrows teach AI the distance between words

What RoPE is

Inside the AI models behind chatbots, attention compares words with each other, but on its own it has no idea which word came where. Position has to be added somehow. RoPE, short for rotary position embedding, does it in an unusual way. It adds nothing to the words at all. Instead, it rotates part of each word's numbers by an angle that depends on where the word sits: a small turn for an early word, a bigger turn for a later one.

What the video shows

The video pictures each word as an arrow that RoPE turns according to its position. When two words are compared, only the gap between them matters. And stretching this rotation is how many AI models got longer memories.

One clarification: "memory" here means the context window, the amount of text a model can take in at once. It is not about remembering you between conversations.

An everyday example

Imagine the hour hand of a clock. Between 2 o'clock and 5 o'clock it turns through three hours. Between 7 and 10, the same three hours, and the same angle. The angle the hand sweeps tells you the gap, not the time you started from.

RoPE works the same way. Imagine a word at position 5 turned 5 steps and a word at position 8 turned 8 steps. When the model compares them, the only position information left is the 3-step difference. Words at positions 105 and 108 look exactly as far apart.

How it works

When attention decides how much one word should listen to another, it compares two lists of numbers: a query (what this word is looking for) and a key (what the other word has to offer). The comparison is a dot product: multiply the matching numbers and add them all up.

RoPE rotates both the query and the key according to their positions, working on the numbers two at a time. Turning two arrows by the same extra amount does not change how well they line up. So the score ends up depending only on the distance between the two words, not on where they sit in the text. Position becomes relative for free, and there is no table of positions that can run out.

How well it holds up

The course trained small models on windows of 64 tokens (a token is a word or a piece of one) and tested them on longer texts. Perplexity scores how surprised a model is by real text, and lower is better:

| Method | 64 tokens | 128 tokens | 256 tokens | |---|---|---|---| | No position information | 48.8 | 52.6 | 57.5 | | Learned position stamps | 38.6 | 108.5 | 181.9 | | RoPE | 44.1 | 50.5 | 84.8 |

RoPE is not magic. It does get worse beyond the length it trained on: at four times that length its score nearly doubled, and there it even trails the model with no position information. At the trained length, a learned table scores better. But over the same stretch the learned table's score grew almost fivefold, so RoPE degrades far more gracefully.

Why it matters

When a company announces that it has extended a model's context window, to 128K tokens for example, the method behind it is almost always a rescaling of this rotation. That is why a context limit can be moved but does not disappear: stretching the rotation shifts the limit to a new place rather than removing it.

RoPE also shows a useful design idea. Rather than teaching a model a separate label for every position, it builds the notion of distance straight into the comparison the model already makes.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.