Glitch tokens: the usernames that made AI models misbehave

What a glitch token is

A language model reads text as tokens: chunks of text, each with its own entry in a fixed list, a bit like words in a dictionary. For every entry, the model keeps a row of numbers that stands for what that chunk means. Those numbers start out random. They only become meaningful because they get adjusted, a little at a time, every time the chunk turns up while the model is learning.

A glitch token is an entry that made it into the list but almost never turned up during learning. Its numbers are still close to their random starting values. The model is holding a symbol it never really learned, so when that symbol appears, nobody can predict what it will do.

What the video shows

The video explains that some usernames got their own piece in an AI's dictionary but almost never appeared in the text the AI learned from. Their meaning stayed random noise, so the AI acted bizarrely whenever it met them.

The famous case

The best-known example is SolidGoldMagikarp, along with a set of similar strings. When GPT-2 and GPT-3, two earlier language models from OpenAI, met them, things got strange. The models refused to repeat the string back, replied with something unrelated, and sometimes even insulted the user.

The explanation turned out to be ordinary. Those strings were Reddit usernames. They were common in the text used to build the tokenizer (the program that decides which chunks get their own entry), so they earned a token of their own. But they were rare or missing in the text used to train the model itself. The entry existed; the meaning never arrived.

An everyday example

Imagine a phrasebook put together by one team, working from a big pile of forum posts. A traveller then learns the language from a completely different pile of books. A few entries in the phrasebook are forum nicknames that never once appear in the traveller's books. When someone says one of them out loud, the traveller recognises it as a real entry but has no idea what it means. Whatever comes out next is improvised.

How it happens

The key detail is that the tokenizer and the model are trained separately, and they can learn from different text:

• Step 1, the tokenizer. It scans its text and gives a token of its own to strings that show up often.
• Step 2, the model. It learns what each token means from its own training text.

If a string is frequent in step 1 but almost absent in step 2, it gets a place in the list without ever getting a meaning. The model's reaction to it is then set by the random numbers it started with, not by anything it learned.

Why it matters

Glitch tokens look like a spooky bug, but they are a plain side effect of how these systems are assembled. They show that every entry in a model's vocabulary is only as good as the training text behind it. When the text used to build the tokenizer and the text used to train the model differ, some entries can end up with no learned meaning at all, and those become blind spots nobody planned.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.