Embedding table: the plain lookup list hiding behind AI's embeddings

What an embedding table is

An embedding table is a grid of numbers with one row for every token a model knows. A token is a small piece of text; in the course's example, it is a single letter. When the model needs a token, it takes that token's ID number and fetches the matching row. That row of numbers is the token's embedding.

There is no hidden dictionary of meanings and no special geometry engine here. It is a lookup, like finding a line in a spreadsheet. The only unusual part is who writes the rows: nobody does. They start as random numbers, and training adjusts them a little at a time, together with every other number in the model.

What the video shows

Every piece of text has its own row of numbers, and the AI simply looks that row up. No person types those numbers in. Training fills them, and as a result letters that behave alike, such as "c" and "k", end up with rows that look alike.

An everyday example

Imagine a cloakroom with numbered hooks. You hand over ticket 11 and get back whatever hangs on hook 11. The attendant does no arithmetic; the number just says where to look. An embedding table works the same way: the token's ID is the ticket, and the row of numbers is what hangs on the hook.

The twist is what hangs there. In a cloakroom, guests bring their own coats. In an embedding table, every row begins as random noise and is reshaped during training until it is useful for predicting what comes next.

How it works

In the course, a model learns to guess the next letter of a name. The first version is plain counting: a table recording how often each letter follows each other letter. Then the same job is rebuilt as a small neural network.

To feed a letter into the network, you write it as a list of 27 slots (26 letters plus one marker for where a name starts or ends), all zeros except a single 1 for that letter. This is called a one-hot vector. Multiplying it by a grid of weights, the adjustable numbers inside the model, has a neat effect: every zero cancels its row and the single 1 keeps exactly one row. So the multiplication is really a lookup, and real systems skip the math and fetch the row directly.

Training starts from random numbers and is told only one thing: make the correct next letter more likely. After 3,000 steps, the network has rebuilt the counting table almost exactly. Across all 729 cells, the biggest difference is 0.004. After an "a", for example, both the counts and the network say the name ends there about 19% of the time.

A common misconception

"Embedding space" is often described as a map of meaning. The course is blunter: it is a set of coordinates the network learns because they make its own job easier. Some patterns do appear, like "c" sitting near "k", but only where they helped the predictions.

Why it matters

The same kind of table sits at the entrance of the bigger language models later in the course. And if the network only reproduces the counts, why bother? Because counting has run out of road here, while the network can be extended: give it more letters to look at and it keeps improving, which is where the chapter goes next.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.