Contrastive learning: how AI learns which sentences mean the same

What contrastive learning is

Contrastive learning is a way of training an AI model on pairs. You show it two things that belong together, such as a question and its answer, along with things that do not. Training pulls each true pair closer together and pushes the mismatches apart. After enough pairs, the model can turn any sentence into a list of numbers (an embedding) where "close" really does mean "similar in meaning". That is exactly what you need to search by meaning instead of by exact words.

What the video shows

The video sets up a contest. A big AI trained to predict text is asked to judge how similar sentences are, and it does worse than simply counting the words two sentences share. Then a model five times smaller, trained to pull matching pairs together, takes the same test and does far better.

The title's "AI that chats" is shorthand for a language model trained to predict text; the AI behind search is one trained for similarity.

An everyday example

Imagine hiring an assistant for a hardware shop. One candidate has read every book in the library and can finish any sentence you start. The other has spent a month on one drill: a customer question on one card, the product that solves it on another, with a few wrong products mixed in. Which one sends a customer to the right shelf? The second, most likely. Knowing a lot about language is not the same skill as matching things by meaning.

How it works

The course runs the contest on a standard test in which people rated how similar 1,379 pairs of sentences are. Each method is scored on rank correlation: how well its ordering of the pairs, from most to least alike, agrees with the human ordering, where 1 would be perfect. A second test builds an index of 200 sentences and checks how often the model's top pick is the true partner.

The big model is BERT, a well-known language model with about 109 million adjustable numbers. The obvious way to get one list per sentence from it is to average the lists it produces for each word. The pair-trained model is MiniLM, with about 22.7 million.

| How each sentence becomes numbers | Agreement with people | Top pick is right | |---|---|---| | Counting shared words, no model | 0.55 | 89% | | BERT, averaging its per-word lists | 0.47 | 84% | | BERT, its special summary token | 0.20 | 67% | | MiniLM, trained on pairs | 0.82 | 92% |

The big model is not broken. It was trained to guess hidden words, and nothing in that goal asks two sentences with the same meaning to land near each other. MiniLM was trained with precisely that goal, so its closeness scores mean what you want them to mean.

A common misconception

Many people assume you can take any big language model, cut off its last step, and use what comes out as a sentence embedding. The course's rule is the opposite: an embedding model is a different model, trained for a different goal, and usually much smaller. Tutorials still suggest using BERT's special summary token, an extra slot added to every sentence that BERT was pretrained to use for a sentence-level task. In this test it came last by a wide margin.

Why it matters

Search by meaning, matching a sentence to its paraphrase, and pairing a question with the passage that answers it all depend on good sentence embeddings. The course calls this recipe the origin of the whole embedding-model industry. Picking the wrong kind of model quietly makes all of those jobs worse, and nothing crashes to warn you. The rule of thumb: if you need similarity, use a model that was trained on similarity.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.