Perplexity: how many options an AI is really torn between

What perplexity is

When a language model guesses the next letter or word, it spreads its bets: it gives a probability to every possible option. Perplexity sums up how unsure those bets are, in a unit anyone can picture: the number of equally likely options the model is effectively choosing between.

A perplexity of 27 means "I am as lost as someone picking at random from 27 options". A perplexity of 1 would mean the model is always completely sure, and right. So lower is better.

What the video shows

The video puts two numbers side by side. A model that guesses the next letter blindly, treating all 27 options as equally likely, scores exactly 27. Give it one letter of context and its perplexity drops to about 12.

In the course, those 27 options are the 26 letters plus a mark for the end of a name, because the model is learning to write names. The context comes from simply counting which letter tends to follow which.

An everyday example

Imagine a game where you must guess the next letter of a hidden name. With no clue at all, you are rolling a fair 27-sided die. Now someone tells you the previous letter was "q". You still cannot be certain, but you would bet heavily on "u", and your real range of choices has shrunk. Perplexity measures that shrinking. A score of about 12 means the model is as uncertain as someone rolling a fair 12-sided die: clearly better than blind guessing, and still far from confident.

How it is calculated

Training an AI produces a loss, a penalty that grows when the model gave little probability to what really came next. For the name model, the average loss on names it never trained on is 2.4546. On its own, that number tells a person almost nothing.

The reason is that the loss lives on a logarithmic scale, a squashed scale like the one used for earthquakes. Perplexity simply undoes the squashing: raise the number e (about 2.718) to the power of the average loss. For the name model, e to the power of 2.4546 is about 11.6. For the know-nothing model, the loss is 3.2958, and e to that power is exactly 27. That is why people quote perplexity and not the raw loss: it comes back as a count of options.

Two traps when reading it

• One impossible event breaks it. If the model gives zero probability to something that then happens, perplexity shoots to infinity. The usual patch for it is called smoothing.
• It depends on how the text is cut up. Scoring per letter, per word piece or per whole word gives very different numbers for the same model. Only compare perplexities between models that split text into the same pieces. The fair alternative is bits per character.

A quick sanity check helps too. The know-nothing score is always the number of possible symbols, so you can see how far below pure guessing a model really sits.

Why it matters

Perplexity is a figure you will often see when language models are compared. Reading it well is simple once you know the trick: it is the size of the fair die the model is effectively rolling at each step. A smaller die means a better model, as long as the comparison is a fair one.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.