Bigram model: an AI name-maker built from a table of letter pairs

What a bigram model is

A bigram is simply a pair of neighbours: two symbols that sit side by side. A bigram model is the most basic language model you can build. It looks at the last symbol written and nothing else, and asks one question: in everything I have seen, what usually came after this one?

There is no neural network, no long training run and no clever maths. There is just a table of counts.

What the video shows

The video builds one from a list of 32,033 real names. It counts which letter follows which, then writes brand-new names by rolling weighted dice, one letter at a time. The results sound right in small pieces but ramble on and on, because the model only ever remembers a single letter.

How it works

1. Pick the alphabet. Here it is the 26 letters plus one extra symbol that marks where a name starts and where it ends. That makes 27 symbols, and the marker is how the model learns that names have a beginning and a stopping point.
2. Count the pairs. Draw a 27-by-27 grid. For every pair of neighbours in every training name, add one to the matching cell. The grid holds 729 numbers. (In the course, 25,626 of the 32,033 names are used for counting; the rest are kept aside to check the model later.)
3. Turn counts into chances. Divide each row by its own total so the row adds up to 100%. For example, after the letter "a" the most likely thing is that the name simply ends (about 19% of the time), followed by "n" (16%) and "r" (almost 10%).
4. Write a name. Begin at the start marker, roll dice weighted by that row, jump to the row of the letter you got, and repeat until the end marker comes up.

Fitting the whole model takes two lines of code. And it is not a rough rule of thumb. Dividing each count by its row total gives exactly the maximum-likelihood answer: the table that makes the real names as probable as possible.

An everyday example

Think of a line of dominoes where each tile only has to match the one before it. Every single joint can be perfectly legal, and the line can still wander off in any direction, because no tile knows what shape the whole line is making.

That is exactly how the bigram model fails. One of the names it produces is "momakurailezitynn". Every neighbouring pair of letters in it really does appear in real names. There are just seventeen letters in a row, and nothing tells the model that it has gone on far too long. The course sums this up as locally plausible, globally nonsense.

Why it matters

The bigram model is the honest starting point, the baseline that every fancier language model has to beat. It shows the basic loop of text generation (predict the next symbol, pick one, repeat) with nothing hidden inside. Its weakness also points straight at what better models need: a memory that reaches back further than one step, so the next letter can depend on the whole word so far.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.