Contextual embeddings: how AI tells a river bank from a money bank

What contextual embeddings are

An embedding is a list of numbers that stands for a word or word piece. In simpler models, that list comes from a lookup table: one row per word, always the same. This has a built-in limit. The word "bank" gets exactly one list, whether the sentence is about a river or about money, because a lookup by word number cannot take anything else into account.

A contextual embedding removes that limit. Instead of reading the list out of a table, the model computes it from the whole sentence, so every use of a word gets its own list. The idea arrived with a model called ELMo in 2018 and became standard with another language model, BERT, the same year.

What the video shows

A lookup table gives the word "bank" one fixed meaning. Contextual embeddings rebuild the word from its sentence, so the river bank and the money bank drift apart on their own.

An everyday example

You never read "bank" in isolation. In "we had a picnic on the bank", you already picture grass and water. In "the bank closed my account", you picture a counter and paperwork. The surrounding words settle the meaning before you even notice there was a choice. Contextual embeddings do the same thing with numbers: the words around "bank" reshape its list, step by step.

How it works, measured

The course takes two sentences. One describes someone sitting by a river, watching the water. The other describes someone depositing a cheque at a bank on a street corner. It then compares the two "bank" lists inside BERT using cosine similarity, a score where 1 means the two lists point exactly the same way and lower numbers mean more different. BERT processes text through 12 stacked stages, called layers.

| Where the "bank" list comes from | River vs money | |---|---| | Lookup table | 1.00 (identical by design) | | Layer 0 | 0.95 | | Layer 4 | 0.56 | | Layer 9 | 0.43 | | Layer 12 | 0.53 |

Meanwhile, two uses of "bank" that both mean a riverside still score 0.87 at layer 9. Same meaning, similar lists; different meaning, clearly different lists.

Two details are worth noticing:

• Layer 0 is already below 1. Before any real processing, the model mixes in where each word sits in the sentence, and "bank" sits in a different spot in each one.
• The last layers drift back together. The final layers are tuned to the model's own training task, so they are often not the best place to take a word's representation from.

Why the meanings separated

Nobody labeled "river bank" or "money bank" anywhere. BERT was trained to guess a hidden word from the words around it. Telling the two senses apart makes that guessing easier, so the separation appeared by itself.

Why it matters

This is the step from one fixed meaning per word to one meaning per use. Since BERT, computing a word's numbers from its sentence, instead of reading them from a table, has been the standard approach. It also gives a practical tip: if you take word representations from a model like BERT, an earlier layer can serve you better than the very last one.

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.