Number tokenization: why AI reads digits in uneven chunks

What number tokenization is

Before a language model reads anything, a program called a tokenizer cuts the text into chunks called tokens, and each chunk is swapped for a number from a fixed list. Numbers in your text get the same treatment as words. The model never sees 12345 as five digits sitting in neat columns. It sees whatever chunks the tokenizer happened to produce.

Number tokenization is simply how those digits get cut up. In the tokenizer the course measures, the honest answer is: unevenly.

What the video shows

The video compares two numbers. 1234 becomes "123" and "4". 12345 becomes "123" and "45". The digits of the two numbers don't line up with each other, so adding them is harder than it looks.

An everyday example

When you add numbers on paper, you line them up from the right: ones under ones, tens under tens. Now imagine a friend hands you each number already cut into paper strips, with the cuts in a different place every time.

That is the model's situation. Look at the chunk "123". Inside 1234, it stands for one thousand two hundred and thirty. Inside 12345, the very same chunk stands for twelve thousand three hundred. Same piece, different value. Before it can add anything, the model first has to work out where each digit really sits, and that puzzle changes with every new pair of numbers.

A few more cuts from the course:

| Number | Pieces the model sees | |---|---| | 2024 | "202" + "4" | | 3.14159 | "3" + "." + "141" + "59" | | 1000000 | "100" + "000" + "0" |

Even an ordinary year is split in an odd spot. And a million comes out in three pieces that don't match the way we group it with commas, 1,000,000.

How it gets fixed

Some newer tokenizers tackle this head-on: they always cut digits into consistent blocks of three, precisely to take this obstacle away. According to the course, models trained with those tokenizers are measurably better at arithmetic. The model has not suddenly become smarter in some deep way. The numbers simply reach it in a more predictable shape.

It starts before the maths

It is tempting to think a model fumbles sums because it is "bad at maths" in some deep way. Part of the problem starts earlier, in how the numbers were chopped before the model ever saw them. A person handed numbers already cut into uneven strips would slow down too.

Why it matters

This helps explain a puzzle many people run into: a model can write a fluent paragraph and still slip on a simple sum. When an exact figure matters, such as a total, a date or a price, it is worth checking the model's arithmetic rather than taking it on trust. And if you ever compare models on maths, remember that the way they cut numbers is part of the story.

Learn it step by step in Chapter 7 of our free course AI From Scratch: Build a BPE Tokenizer: Why Your Model Can't Count the R's.