Cosine similarity: comparing where two arrows point, not how long

What cosine similarity is

AI systems turn letters, words and documents into lists of numbers. Picture each list as an arrow. Cosine similarity asks one question about two arrows: do they point the same way? It ignores how long they are.

The score runs from 1 (exactly the same direction), through 0 (unrelated, at a right angle), down to −1 (opposite directions). Under the hood it is the dot product, a way of multiplying two lists position by position and adding up the results, with the arrows' lengths divided out so that only the angle is left.

What the video shows

Cosine similarity compares where two arrows point and ignores their length. In the big spaces that AI systems work in, two random arrows almost never score above 0.1. So a score of 0.4 is a real match. It does not mean the two things are "40% alike".

An everyday example

Imagine two hikers who both set off north-east. One walks 2 kilometers, the other walks 10. Their trips are very different in length, but their heading is identical, so cosine similarity gives them a perfect 1. A third hiker heading south-west would score −1.

Why ignore length at all? Because in an embedding (the list of numbers a model learns for each token), the length of an arrow often reflects how frequently that token showed up in training rather than what it means. Direction is the fairer comparison.

How to read a score

• Compare it with random pairs, not with 100%. A score is only high relative to what unrelated items get in the same space. In a big space where unrelated pairs almost never pass 0.1, a 0.4 stands well clear of the noise. The bar depends on the space, so check what random pairs score there first.
• Normalize once, then it is just a dot product. Real systems usually shrink every arrow to length 1 when they store it. After that, cosine similarity and the plain dot product give the same number, which is quick to compute.

What it finds inside a real embedding table

The course trains a tiny table that gives each letter a row of 10 numbers, learned only from guessing the next letter of names, and then uses cosine similarity to find each letter's closest neighbor. Some results match the folklore:

• "c" is closest to "k" (0.60) and "i" to "y" (0.54): letters that swap freely in names.
• "j" is closest to "z" (0.65): two rare letters that tend to start names.
• The symbol that marks where a name starts or ends is close to nothing (0.08 at best), because it marks a position, not a sound.

Other results do not match. The closest neighbor of "a" is "h", not another vowel. On average, vowels score 0.19 with each other and about 0 with consonants: a real pattern, but a small one. To judge it, the course did not compare it with a perfect score but with 2,000 random groups of five letters: 58 of them separated at least as cleanly. So the pattern is probably not luck (about a 3% chance), yet it is nothing like a neat island of vowels.

Why it matters

Cosine similarity is the standard tool for asking which items sit close together. Reading it well takes two habits: judge a score against the noise level of its space, and remember that an embedding only holds the structure that helped the model lower its error. It is a set of learned coordinates, not a tidy map of meaning.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.