Subword tokens: the middle ground between whole words and letters

What subword tokens are

Before a language model can read your message, the text has to be cut into pieces, and each piece gets a number. Subword tokens are how most modern models do that cutting: the pieces sit between single letters and whole words. A common word gets a piece of its own. A rare word is broken into a few smaller pieces. And no text is ever left without a number.

The idea is easiest to grasp by looking at the two obvious alternatives. Both fail, and the way they fail explains the design.

What the video shows

The video weighs the two simple options. Whole words leave any new word without a number. Single letters make every text far longer. Then it shows the middle path AI actually uses: subword pieces, where common words stay whole, rare ones are split, and nothing is unknown.

Why not whole words?

• The list gets huge. English alone has hundreds of thousands of word forms, and every single one would need its own entry. On top of that, every time the model writes a word, it has to give a score to every entry on that list.
• New words have no number. A word the model never met in training, say a fresh slang term or an unusual surname, simply does not exist for it. The usual patch is to swap every unknown word for one generic "unknown" marker, which throws the actual word away.
• "Word" is not even a clear idea. Chinese and Japanese are written without spaces between words, and German glues nouns together into very long compounds.

Why not single letters?

Letters solve the unknown-word problem, since around a hundred symbols can spell out English. But texts get four to five times longer: a 1,000-word document is roughly 5,000 characters. That is costly, because the work done by attention, the part of a model where every piece looks at every other piece, grows with the square of the length. Twice as long means about four times the work. And a lone letter means almost nothing, so the model spends effort gluing words back together.

An everyday example

Imagine designing a keyboard. One with a key for every English word would need hundreds of thousands of keys, and it could never type a word invented tomorrow. One with only letter keys is compact, but every message takes many more presses. The sensible design is letter keys plus shortcut keys for the most common words and word parts. That is what subword tokens are: common things get a shortcut, and everything else can still be spelled out.

Who decides the pieces?

Nobody does, at least not by hand. The tokenizer (the program that does the cutting) is trained on large amounts of text, and it learns which pieces deserve their own number by counting which ones keep turning up side by side. Because the smallest pieces are single bytes, the basic units computers store text in, there is always a way to write any input.

Why it matters

This one design choice explains a lot about how chatbots behave. It is why usage limits and prices are counted in tokens rather than words, why an unusual name can use up several tokens, and why a model can meet a word it has never seen and still take it in. The exact recipe that picks the pieces, byte-pair encoding, is a story of its own in the same chapter.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.