ALiBi: the distance penalty that lets AI read past its training length

What ALiBi is

Every chatbot-style AI model relies on attention, the step where words look at each other, and attention has no built-in sense of word order. So every model needs some way to add position. Most methods give each word a kind of position stamp. ALiBi skips the stamps entirely. It simply lowers the attention score between two words by an amount that grows with the distance between them. Nearby words get a small penalty; faraway words get a bigger one.

There is one more ingredient. A block of attention runs several heads side by side, each one a separate looker that decides for itself what to focus on. ALiBi gives every head its own penalty rate, called its slope.

What the video shows

The video starts from a common weakness: many AI models struggle with texts longer than the ones they trained on. ALiBi simply makes words quieter the farther away they are. Trained on windows of 64 tokens (a token is a word or a piece of one), it did even better on windows of 256.

An everyday example

Imagine a crowded party. Voices fade with distance: the person beside you is loud, someone across the room is faint. Now imagine a group of listeners with different hearing. One mostly catches the people right next to them. Another follows conversations from the whole room. Together, they cover both the close and the far.

That is ALiBi's per-head slope: a steep slope for a head that focuses nearby, a gentle one for a head that listens far. And if the party moves to a bigger hall, nothing breaks. The fading rule works at any distance.

How it works

1. Score. Attention computes a score for how much each word should listen to each earlier word.
2. Penalize. ALiBi subtracts a penalty in proportion to the distance between them: twice as far, twice the penalty.
3. Vary by head. Each head uses its own slope, so some heads stay local and others reach further.

There are no position embeddings at all. So there is no table of positions that can run out, and no untrained entry waiting at some far-off position: a distance of 200 gets its penalty from the same rule as a distance of 20.

The result

The course trained small models on those 64-token windows, then tested them on longer texts. Perplexity scores how surprised a model is by real text, and lower is better:

| Method | 64 tokens | 128 tokens | 256 tokens | |---|---|---|---| | ALiBi | 44.95 | 43.51 | 42.49 | | Learned position stamps | 38.63 | 108.47 | 181.94 |

Two things stand out. At the length it trained on, ALiBi is not the winner: learned stamps score better there, and in the course's full comparison the other two position methods were slightly ahead too. But past that length, learned stamps collapse while ALiBi keeps improving. The course's explanation is short: the penalty is defined at any distance, so every head keeps doing exactly what it learned to do.

A common misconception

It is tempting to read the table as "ALiBi is the best method". The honest reading is narrower. In this experiment it was the sturdiest when texts grew beyond the training length, not the strongest at the length it practised on. Which one you want depends on whether your model will meet longer texts than it trained on.

Why it matters

The course calls ALiBi both the simplest and the strangest result in its comparison. It is a reminder that a tiny, plain rule can be sturdier than a clever one, and that how a model encodes position decides what happens when it meets a text longer than any it trained on.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.