Bits per character: the fair way to compare language models

What bits per character is

Bits per character is a way to score a language model that does not depend on how the text was chopped up. It asks: on average, how much does it cost the model to predict each character of the text, meaning each letter, space or punctuation mark? The cost is counted in bits, the basic unit of information, where one bit is the answer to a single yes-or-no question. Lower is better.

Why it is needed

Language models do not have to read text letter by letter. They read tokens, pieces of text chosen by a tokenizer (the rule that cuts text up). Tokens can be single characters, chunks of words or whole words. The usual score, perplexity, is a price per token. So if you change the size of the tokens, you change the price, even when the model itself is no better and no worse.

What the video shows

The video makes that point directly: scoring an AI per piece of text depends on how big the pieces are. Measure the cost per letter instead, and the comparison becomes fair, and the ranking of the models changes completely.

The course tests this on one simple counting model, trained on the same English text each time. Only the way the text is cut changes. "Word pieces" are built by a method called BPE, which repeatedly merges frequent pairs of symbols into bigger pieces:

| How the text is cut | Perplexity | Bits per character | |---|---|---| | Single characters | 12.45 | 3.64 | | Word pieces, 512 merges | 47.21 | 2.64 | | Word pieces, 2,048 merges | 313.20 | 2.43 | | Whole words | 35.26 | 2.23 |

By perplexity, characters look best and the 2,048-merge version looks worst, roughly 25 times worse. By bits per character the order flips: whole words win and characters come last. The 2,048-merge version, which looked 6.6 times worse than the 512-merge one, is actually the better of the two.

An everyday example

Supermarkets solved this problem long ago. A small jar of coffee can cost less than a big bag, but that tells you nothing about which one is the better deal. So the shelf label also shows the price per kilo, which lets you compare packs of any size fairly. Perplexity is the price per pack. Bits per character is the price per kilo.

How it works

Predicting a whole word is harder than predicting a single letter, so each word prediction costs more. But there are far fewer words than letters to predict. Bits per character adds up the total cost of predicting the whole text, then divides by the number of characters instead of the number of tokens. Every model is now charged for exactly the same thing: the same letters of the same text.

Why it matters

Comparing models that use different tokenizers by perplexity is a mistake the course says turns up constantly, even in published papers. Bits per character is the fair comparison. It also gives a sense of scale. In 1951, Claude Shannon measured it by asking people to guess the next letter of English text, and put English at roughly one bit per character. The best counting model in the table sits at 2.23 bits, which shows how much room there is left for smarter models.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.