The strawberry problem: why AI can miscount the r's in a word

What the strawberry problem is

Ask a language model clever enough to pass a lawyer's bar exam how many r's there are in "strawberry", and it may well answer two. The word has three. It sounds like a question a child could answer, which is exactly why the mistake surprises people.

The popular explanations are that AI is "bad at counting" or "does not really understand". Neither can be tested, and neither is the cause. The real reason is mechanical, and it happens before the model even starts working on your question.

What the video shows

The video shows that AI does not read letters. The word "strawberry" reaches the model as three chunks, each turned into a number, and the r's are hidden inside those chunks. Counting them means guessing at something the model cannot see.

What the model actually receives

Before a model reads anything, a program called a tokenizer (a text chopper) cuts your words into pieces called tokens and gives each piece an ID number. In the course's example, "strawberry" becomes three tokens, "str", "aw" and "berry", with the IDs 496, 675 and 15717.

So the question "how many r's?" really asks: how many r's are tucked inside token 496, plus token 675, plus token 15717? The model only has the numbers. Nothing about the number 15717 spells out b-e-r-r-y. It is being quizzed about something it was never shown.

An everyday example

Imagine a waiter who writes down orders only as dish numbers. Table four orders dish 12 and dish 7. Then the manager asks him: how many eggs are in that order? The waiter never saw the recipes, only the numbers. If dish 12 is the house omelette they serve all day, he probably remembers. If dish 7 is a rare special, he is guessing.

That is the model's position. It can only answer questions about letters from memory of how each chunk is spelled.

How to spot it

The same cause sits behind a whole family of odd slips. Be careful with any task that depends on the letters inside a word:

• Counting letters, like the r's in "strawberry".
• Spelling a word out one letter at a time.
• Reversing a word, writing it backwards.

Models often get these right anyway, because during training they absorbed the spellings of many words. That works well for common words and badly for rare ones. So a model can spell an everyday word flawlessly and still stumble on an unusual name.

Why it matters

The strawberry problem is a handy reminder that a chatbot does not see your text the way you do. When a model trips over something that looks trivial, the cause may be in how the text was cut up, not in how smart the model is. Knowing that helps you predict where it will slip: any job about the letters inside words deserves a double-check. The rest of the chapter shows how the chopping itself works, from subword pieces to the merge rules that decide them.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.