WordPiece: why the most surprising pair beats the most common one

What WordPiece is

WordPiece is a way of building a tokenizer's vocabulary: the fixed list of text chunks, called tokens, that a language model reads. It is the method behind BERT, a well-known language model.

Like byte-pair encoding (BPE), the method the course builds by hand, WordPiece starts from tiny pieces and keeps gluing two neighbouring pieces into a new, longer one until the list is big enough. The two methods differ in one thing only: the rule for deciding which pair to glue next.

What the video shows

The video contrasts the two rules. The usual approach glues together the pair of pieces seen most often. WordPiece prefers pairs that appear together more than you would expect, so two rare pieces can come out on top.

An everyday example

Imagine you want to find the real best friends in a big school. If you simply count how often two students are seen together, the two most popular kids win every time. They are seen with everyone, so of course they are often seen with each other.

A fairer question is: how often are these two together, compared with how often you would expect it, given how much each of them is around anyway? Two quiet students who rarely show up at all, but are almost never apart, can score highest on that question. That is WordPiece's question.

Written English has the same pattern. The letter "q" is fairly rare, yet it is nearly always followed by "u". A rule that rewards that kind of loyalty, instead of raw popularity, captures the idea behind WordPiece.

How the rule works

Both methods scan the training text and look at every pair of neighbouring pieces. What changes is how they score each pair:

| | BPE | WordPiece | |---|---|---| | Question asked | Which pair appears most often? | Which merge makes the training text most likely? | | Popular halves | Help a pair win | Count against it | | Two rare pieces | Win only if their pair is the most frequent | Can win if they stick together more than chance |

"Makes the training text most likely" means the merge that best explains the text the tokenizer has seen. In practice it works as a correction for popularity: a pair gets less credit when its two halves are common on their own, and more credit when they turn up together more often than chance would suggest.

Why it matters

The vocabulary decides how every sentence is cut into pieces before the model reads it. Two tokenizers trained on exactly the same text can end up with different lists of pieces, just because they used a different rule for choosing merges. And since the model only ever sees those pieces, that choice quietly shapes everything it learns.

WordPiece is a neat reminder that "seen most often" is not the only sensible meaning of "belongs together".

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.