Bag of words: why some AI search can't tell who bit whom

What a bag of words is

A bag of words is the simplest way to turn a sentence into something a computer can compare. You note which words appear and throw the order away. "The cat sat on the mat" becomes a little pile: the, cat, sat, on, mat. Two sentences count as similar when their piles share many words. No training and no model, just counting.

What the video shows

The video takes two sentences: "the dog bit the man" and "the man bit the dog". Treated as bags of words, they are identical, because they use exactly the same words. Then comes the uncomfortable part: even a search model trained specifically to judge meaning scores the pair 0.979, almost a perfect match.

An everyday example

Imagine a travel site that matches your search by shared words. You type "flights from Madrid to Paris". A result that says "flights from Paris to Madrid" contains every one of your words, so it looks like a perfect hit. It is the opposite trip.

The old newsroom saying makes the same point: "dog bites man" is not news, but "man bites dog" is. Same words, very different story.

Why it works better than it sounds

A bag of words sounds too crude to be useful, yet it holds up surprisingly well. In the course's test, people rated how alike 1,379 pairs of sentences are. Plain word overlap agreed with those ratings better than BERT, a language model with about 109 million adjustable numbers, used the obvious way: 0.55 against 0.47, where 1 would be perfect agreement. In a search test over 200 sentences, word overlap put the right partner first 89% of the time, against 84% for that BERT setup. Only a model trained specifically on matching sentence pairs pulled clearly ahead, at 0.82.

A likely reason: sentences that mean the same thing tend to share many of their words, and that alone often shows whether two texts are about the same thing.

The blind spot

Order is exactly what the bag throws away, so anything that depends on order is invisible to it. Word overlap gives the two dog sentences a similarity of exactly 1, the highest score possible. And the blind spot reaches further than you might expect: BERT, which does take word position into account, still lands close to 1, and the model trained on meaning gives 0.979.

The course's verdict is blunt: when a task turns on who did what to whom, no cut-off will rescue it. Raising the similarity threshold (the minimum score that counts as a match) does not help, because the backwards sentence scores almost as high as the right one.

How to spot it

Try a quick test on any search or matching system you rely on:

• Take a few real sentences and swap who does what: "the tenant owes the landlord" becomes "the landlord owes the tenant".
• Compare the similarity scores. If they barely move, the system cannot see order.
• Wherever direction matters, such as payments, travel or blame, do not let the similarity score be the final judge.

Why it matters

Word counting is a cheap baseline that a bigger model, used carelessly, may not beat, so it is worth measuring first. Its blind spot is worth knowing too, because it also hides inside more advanced tools, where "very similar" can quietly mean "exactly backwards".

Learn it step by step in Chapter 8 of our free course AI From Scratch: Next-Token Prediction: Embeddings, and What Perplexity Means.