Smoothing: why an AI must never treat the unseen as impossible

What smoothing is

Smoothing is a small correction for models that learn by counting. Before turning counts into probabilities, you give every possible outcome a little extra, so that nothing ends up with a probability of exactly zero. The simplest version, called add-one smoothing, just adds 1 to every count.

What the video shows

The video looks at a model that learned letter pairs by counting them in thousands of names. Any pair it never saw, it treats as impossible. Then a rare name turns up containing one of those pairs, and the model's score breaks completely. Smoothing gives every pair a tiny chance and repairs the score. But, as the video points out, the counting model still cannot generalize: it has no way to reason from the pairs it has seen to the ones it has not.

An everyday example

Imagine a tourist who spends one dry week in a city and concludes that it never rains there. If they bet everything on that, a single shower wipes them out. A sensible visitor says: "I have not seen rain yet, so it is probably rare, but it is not impossible." Smoothing forces the model to be that sensible visitor.

Why a single zero is so destructive

The course's letter-pair table has 27 × 27 = 729 cells, one for every possible pair of symbols. Of those, 113 cells (about 15.5%) never appear in the training names. That does no harm until the held-out names (names kept aside to check the model, never used for counting) land in one of the empty cells. Seven letter pairs in those held-out names do, including "d" followed by "q", "z" followed by "j", and "q" followed by "o" twice.

The model's score, its loss, charges a penalty for every prediction, and the penalty grows as the probability the model gave to what really happened shrinks. At zero probability, the penalty is infinite. So one name out of roughly three thousand is enough to make the loss infinite, and the perplexity built on it too. One surprise, and the whole measurement is useless.

Adding 1 to every count before turning them into probabilities removes every zero, and here it barely changes the model's score at all.

What smoothing cannot fix

The course calls this patch a confession. A person looking at "q" followed by "o" might reason like this: "q" is very often followed by "u", and "o" behaves a lot like "u" in other places, so "q" then "o" is not so strange. The count table cannot think like that. Each of its 729 cells is learned entirely on its own, with no idea that two letters might resemble each other. Smoothing hands out a little probability evenly. It does not make the model any smarter about which unseen pairs are actually likely.

Fixing that needs a model that can learn which symbols are similar. That is the job of learned embeddings, which come later in the same chapter: each symbol gets a short list of numbers, learned from the data, so that letters which behave alike end up with similar lists.

Why it matters

Any model that learns by counting has to decide what to do about things it has never seen. "Never seen" and "impossible" are two very different claims, and mixing them up can wreck a model's score at the worst possible moment: on the new data it was built for.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.