Curse of dimensionality: when everything is almost equally far away

What the curse of dimensionality is

A dimension is one number used to describe something. A spot on a map needs two numbers: how far east and how far north. The lists of numbers AI uses to describe words and sentences often have hundreds or thousands of them. The curse of dimensionality is the strange fact that, with that many numbers, distances stop behaving the way our eyes expect. Everything ends up roughly the same distance from everything else.

What the video shows

The video starts on a flat map, where some points sit close together and others are far apart. Then it jumps to 10,000 dimensions. There, the farthest pair of points is only 6% farther apart than the closest pair. So "the nearest one" means much less than it does on a map.

An everyday example

Imagine comparing people using two traits, such as height and age. Some pairs are almost twins and others are worlds apart. Now imagine comparing them on 10,000 unrelated random traits. Every pair differs on lots of small things, those differences pile up, and every pair ends up about equally different. Nobody is really anyone's "closest match" any more.

The numbers

The course scatters 1,000 random points in a space with more and more dimensions, and compares the farthest pair with the nearest pair:

| Dimensions | How much farther the farthest pair is | |---|---| | 2 | about 1,900 times | | 10 | about 9.9 times | | 100 | about 1.7 times | | 1,000 | about 1.2 times | | 10,000 | only 1.06 times |

On a flat surface, "nearest" is hugely informative. In 10,000 dimensions it tells you very little. The course points out that this is one reason big vector databases (systems that store huge numbers of these number lists and fetch the closest ones to a query) do not insist on finding the exact nearest match.

The good news hidden inside

The way distances bunch together has a useful side: it squeezes chance matches too. The course checks this with cosine similarity, a score for how closely two lists of numbers point the same way, where 0 means unrelated. Between random lists in 768 dimensions, the average score lands almost exactly on 0, and only about 0.2% of pairs stray past 0.1 in either direction. In 100 dimensions the scores spread out more: their typical distance from zero is about 0.1, against about 0.04 in 768. More dimensions, tighter bunching.

So when two sentences score 0.4, they sit far outside anything chance produces. That is why similarity cut-offs somewhere in the 0.3 to 0.7 range can tell real matches apart from noise.

A common misconception

"If distances blur in high dimensions, similarity scores must be useless." The course draws the opposite lesson. The bunching that makes "nearest" fuzzy is the same bunching that keeps random scores pinned near zero, so a score well above zero stands out clearly. The other half of the mistake is reading a score as a percentage: 0.4 does not mean "40% alike". The right question is always: close compared with what chance would give?

Why it matters

Every search-by-meaning system works in these high-dimensional spaces. Knowing the curse explains why "nearest" can be fuzzy, why big systems settle for approximate search, and why a modest-looking score of 0.4 can still be a strong signal.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.