Embeddings: why a puppy lands closer to a dog than to a pizza

What an embedding is

An embedding is a list of numbers that an AI model gives to each word (or to each token, the word pieces from AI 101 #05). Read those numbers as coordinates and every word becomes a point in space. The useful part is where the points end up: words that get used in similar ways land near each other. That is the idea behind the video's question: "puppy" should land near "dog" and far from "pizza", without anyone telling the model so.

The course adds a caution. This is not a dictionary of meanings stored somewhere inside the model. It is a way of placing words so that patterns of use turn into distances you can measure. "Map of meaning" is a handy picture, not a literal description.

What the video shows

The video draws words as points. Similar meanings sit close together. Then it adds a twist: directions seem to mean something too, so the step from "man" to "woman" looks like the step from "king" to "queen". Real models use hundreds or thousands of numbers per word; the video shows only three, so the space can be drawn.

An everyday example

Imagine you move to a new town and know nobody. Just by noticing who sits with whom at the café every morning, you would soon guess who belongs together, without anyone explaining. Embeddings are learned in a similar way: from which words keep showing up near which other words, across huge amounts of text.

How it is learned

One classic method, word2vec, plays a simple guessing game over and over. Here are two words: did they really appear close together in the text, or is this a random pair? Real pairs get nudged closer, fake pairs get pushed apart. After enough rounds, words that keep the same company share a neighbourhood.

The course trained its own small set of embeddings this way on about 17 million words of English Wikipedia, with 100 numbers per word. The nearest neighbours came out like this:

| Word | Closest words it found | |---|---| | guitar | bass, vocals, acoustic, guitars, drums | | three | seven, two, one, five, four | | physics | chemistry, electromagnetism, quantum, theoretical |

No one supplied categories like "instruments" or "numbers"; the groups emerged from the text alone.

A common misconception

The famous demo goes: take "king", subtract "man", add "woman", and you land on "queen". The course tested it on those same embeddings. The point nearest to the result was "king" itself. Even after ruling out the three starting words, which is what the standard test quietly does, "queen" only came fourth.

It is not a one-off. Across more than 4,000 questions of that kind ("Paris is to France as Rome is to what?"), the closest point was one of the input words 99.8% of the time, before that rule removed them. And a shortcut with no arithmetic at all, just returning the word nearest to the third one ("woman" in the demo), kept about three quarters of the standard method's score.

The video shows the classic picture, and directions do carry some meaning. But the effect is far weaker than the one showcase example suggests. Much of what looks like reasoning by analogy is really closeness to one of the starting words, plus a scoring rule that hides the obvious answers. The course's embeddings are small, so its percentages are rough, but it argues the pattern holds at every size.

Why it matters

On its own, a token's ID number says nothing about meaning: number 496 is not "closer" to 497 in any useful sense. An embedding swaps that bare ID for a position, and positions can be compared. That is the first step that lets a model treat "guitar" and "bass" as related instead of as two unrelated labels.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.