Prediction is compression: a good guesser only writes down surprises

What "prediction = compression" means

If you can predict what comes next, it costs almost nothing to store. You mainly need to write down the parts that surprised you. That simple idea links two things that sound unrelated: a language model guessing the next word, and a program that shrinks files.

The link is exact, not a loose comparison. A language model's total error on a text, measured in bits, is the size of the compressed file it could turn that text into. A lower error and a smaller file are the same number, written in two different units.

What the video shows

The video makes one point and makes it plainly: if you can predict what comes next, you only need to record the surprises. So a language AI's error, measured in bits, is exactly the size of the file it could compress text into.

An everyday example

Imagine telling a close friend about your commute. If it went as usual, "normal day" covers it, because they could have predicted the rest. If a parade closed the main road, that is the part that takes words. A friend who knows your routine lets you say very little. A stranger needs every detail.

A compressor works the same way. The better it can guess what comes next, the fewer bits it spends on the predictable parts, and the more of its budget goes on the genuine surprises.

How it works

A bit is one yes-or-no answer. When a model gives the next token (a word or a piece of a word) a probability, that probability sets how many bits it costs to write that token down. Likely tokens are cheap and unlikely ones are expensive:

| Chance the model gave the real next token | Cost to store it | |---|---| | 1 in 2 | 1 bit | | 1 in 8 | 3 bits | | 1 in 1,024 | 10 bits |

A technique called arithmetic coding can turn any model's predictions into a working compressor that pays those costs, token after token. Add them all up and you get the size of the compressed file. Divide that same total by the number of tokens and you get the model's error score, its cross-entropy, counted in bits per token. One quantity, read two ways.

There is also a hard limit. In 1948, Claude Shannon showed that no code can, on average, be shorter than the entropy of its source, which is the true amount of unpredictability in the data. A better predictor gets closer to that floor.

A common misconception

"A model that only predicts text has not really learned anything." Compression is the cleanest answer to that. To squeeze text into fewer bits, a model has to capture whatever produced the text in the first place, and there is no shortcut that lowers the bits without doing so. That is also why a falling error score means something even when there is no particular task in mind.

Why it matters

This is why "bits per byte" (how many bits the model needs for each byte of raw text; a plain English letter takes one byte) appears next to perplexity in reports about language models. It is also why the rankings for lossless text compression (shrinking text so it can be restored perfectly, letter for letter) are led by neural language models. Getting better at predicting and getting better at compressing are one and the same achievement.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.