Positional encoding: how an AI knows which word came first

What positional encoding is

Attention, the core of the AI models behind chatbots, lets each word look at the other words and pull in what it needs. But it has a blind spot: it does not know where anything is. Shuffle the words of a sentence and each word's result comes out exactly the same, just in a different place. To attention on its own, "dog bites man" and "man bites dog" are made of the same ingredients.

Positional encoding is the fix. It adds information about each word's place in the text, so the model can use word order.

What the video shows

The video starts from that blind spot: attention alone ignores word order, so shuffling a sentence changes nothing. Positional encoding stamps each word with its place. Then comes the catch: stamps the model learned during training stop working beyond the length of text it trained on.

One honest detail. Chatbot-style models are not completely blind, because each word may only look back at the words before it, and that leaks a little order. The course's model with no position information at all still trained. But among the words it can see, every arrangement looks alike, which is why real models add position on purpose.

An everyday example

Imagine a cinema whose ushers have only ever worked in a hall with 64 rows. They know exactly where row 12 or row 50 is. Hand them a ticket for row 100 and they are lost. The ticket is perfectly valid; they simply never learned where it leads.

That is what happens to the simplest kind of position stamp.

Two classic ways to stamp a position

• Learned position embeddings. Every position gets its own list of numbers, added to the word, and the model tunes those lists while it trains. A position that never appeared in training keeps the random values it started with.
• Sinusoids, the choice in the original 2017 design. Each stamp is calculated from waves (sines and cosines) running at a range of speeds, so a stamp exists for any position. The hope was that this would carry over to longer texts.

The course trained small models on windows of 64 tokens (a token is a word or a piece of one), then tested them on 64, 128 and 256. Perplexity scores how surprised a model is by real text, and lower is better:

| Position method | 64 tokens | 128 tokens | 256 tokens | |---|---|---|---| | None | 48.8 | 52.6 | 57.5 | | Learned embeddings | 38.6 | 108.5 | 181.9 | | Sinusoids | 43.0 | 95.3 | 152.3 |

Learned embeddings win at the trained length and then fall off a cliff. Sinusoids fail too: the stamp for position 200 exists, but the model never learned how to read it that far out. At 128 and 256 tokens, both even do worse than the model with no position information at all. Two newer designs, RoPE and ALiBi, hold up much better than these two; they are explained separately.

Cannot, or never learned?

The course draws a distinction worth keeping. A model that cannot represent something has a design problem. A model that never practised something has a training problem, and the second one is the failure that usually bites. Learned stamps are perfectly able to describe position 100. Nothing in training ever showed the model what position 100 means.

Why it matters

Word order carries meaning, so every language model needs some way to know it. And the way position is added decides what happens when a text is longer than anything the model practised on, which is exactly the situation where the simplest stamps break down.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.