word2vec: how a yes-or-no guessing game taught computers about words

What word2vec is

word2vec is a method from 2013 that gives every word a list of numbers, called a word vector or embedding, so that words used in similar ways end up with similar lists. Its key move was to skip building a full language model (a system that predicts the next word) and go straight for the part that was proving most useful: the table of word vectors itself.

What the video shows

word2vec learns about each word from the words around it. Pairs that really appear together get pulled closer; random pairs get pushed apart. Nobody tells it which words are instruments, yet "guitar" ends up right next to "bass".

An everyday example

Imagine a quiz built from a huge pile of books. The quizmaster shows you two words. Sometimes they really sat near each other in a sentence, like "strum" and "guitar". Sometimes the second word was picked at random, like "guitar" and "invoice". Your only job is to answer "real pair" or "fake pair".

To win this game over millions of rounds, you would end up keeping a mental note on every word: what kind of company it tends to keep. Words that show up around the same neighbors would get similar notes. That is what word2vec's lists of numbers turn into.

How it works

The popular version is called skip-gram with negative sampling. In plain words:

1. Take a word from the text and one of its real neighbors.
2. Score how well their two lists match, using the dot product: multiply the lists position by position and add up the results. Nudge both lists so that score goes up.
3. Draw a few random words as fake neighbors, and nudge those scores down.
4. Repeat across the whole text.

Each step is a small yes-or-no question about a handful of pairs. It never has to score every word in the vocabulary, which is what a language model must do at every position. That shortcut is why, in 2013, learning from billions of words became practical.

A related method, GloVe, reaches similar vectors from the other side: it starts from one big table of how often each pair of words appears together and squeezes it down. The course's honest summary of both is that they are counting, compressed.

What the result looks like

The course trains word2vec on about 17 million words of English Wikipedia, keeping the 71,290 words that appear at least five times and giving each one 100 numbers. The closest neighbors make sense:

• guitar: bass, vocals, acoustic, guitars, drums
• three: seven, two, one, five, four
• physics: chemistry, electromagnetism, quantum, theoretical

No one supplied a category for musical instruments or for numbers. Those groups came out of word company alone.

Why it matters

word2vec showed that useful word vectors could come cheaply from raw text, with no labels at all. It is also behind the famous word-arithmetic demo, "king − man + woman = queen", which deserves its own careful look because it is weaker than it seems. One caution: the course's vectors are small. Published models learned from billions of words, not 17 million, so treat these neighbor lists as an illustration, not a benchmark.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.