Byte-pair encoding: the 1994 compression trick that splits text for AI

What byte-pair encoding is

Byte-pair encoding, or BPE, is the recipe that decides how a language model cuts text into pieces called tokens. The rule fits in one sentence: find the pair of neighbouring pieces that shows up most often, glue it into one new piece, and repeat. Do that thousands of times and you end up with a list of pieces where common words are whole and rare words are split into chunks.

A short history

BPE was not invented for AI. In 1994 Philip Gage published it as a way to make files smaller: keep replacing the most common pair of neighbouring bytes with a single byte the file does not use. Twenty-two years later, in 2016, the researchers Sennrich, Haddow and Birch reused it for machine translation, to cope with words the system had never seen. Today essentially every large language model reads text this way.

What the video shows

The video shows byte-pair encoding at work: it keeps gluing together the pair of pieces that appears most often. Run on English text, after 12 rounds of gluing it has found the word "the" all by itself, with no dictionary to tell it.

How it works, step by step

1. Start from bytes. Turn the training text into bytes, the basic units computers store text in. A byte has 256 possible values, so the starting list of pieces has 256 entries.
2. Count pairs. Walk through the text and count how often each pair of side-by-side pieces occurs.
3. Merge the winner. Give the most frequent pair a brand-new ID and swap it in everywhere. The list of pieces grows by one, and the text gets shorter.
4. Write it down and repeat. Record which pair became which ID, in order. That ordered list of merges is the whole tokenizer: it is all you need to cut up new text later.

In the course, run on 151,191 bytes of English, merge number 12 is " the " with a space on each side. Nobody told it that "the" is a word. It is simply one of the most common things in English text. As early as merge 3, it had even rediscovered a piece of UTF-8, the standard way computers store characters as bytes.

An everyday example

Imagine a student taking fast lecture notes. They notice they write "th" constantly, so they invent a squiggle for it. Soon "squiggle plus e" is everywhere, so "the" gets its own mark too. Over a term, the notes fill up with private shortcuts for the most common letter groups. That student has just done byte-pair encoding by hand.

A tiny worked case: imagine the whole text is "low low lower". The pairs "l+o" and "o+w" each appear three times, a tie, so just pick one: glue "l+o" into "lo". Now "lo+w" appears three times, so glue that into "low". After two rounds, "low" is one piece, and "lower" is "low" plus "e" plus "r".

Why it matters

BPE explains why models see text the way they do. Common words become single pieces because they are common, not because anyone decided so, and rare words get split. It also shows that some of AI's core machinery is old, short and understandable: the course builds a working trainer in a few dozen lines.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.