Rounding error: why adding the same numbers can give two answers

What rounding error is

A computer keeps every number with a limited number of digits. So each time it adds two numbers, it rounds the exact result to the nearest number it is able to store. One rounding is tiny. A big calculation does millions of them, and the small losses pile up.

The surprising part is that the order of the additions matters. On paper, (a + b) + c always equals a + (b + c). On a computer it does not always, because each step rounds at a different moment. In math terms, computer addition is not associative, which simply means that the way you group the steps can change the answer.

What the video shows

The video adds up the same million numbers twice, in two different orders. The two totals come out about fifteen apart, and neither of them is the right answer.

An everyday example

Imagine keeping a running total in a notebook where every number must fit in four digits. At 12.34 you can still write the cents. Once the total passes 1,000, only whole numbers fit, so 1,234 plus 0.4 gets written down as 1,234 again. Add 0.4 a hundred times, one at a time, and the total never moves, even though 40 went in. Add the hundred 0.4s together first, which gives 40, then add that, and the notebook correctly reads 1,274.

That is what happens inside a long sum. The bigger the running total, the coarser the steps the computer can record, so each small number you add is partly or completely swallowed by the rounding. Change the order or the grouping, and you change how much gets lost.

The experiment in the course

The course adds one million random numbers between 0 and 1, stored as 32-bit numbers:

• Smallest first: 500,166.41
• Largest first: 500,151.31
• The exact total: 500,159.26

The gap is about 15, and both results miss the truth: smallest-first lands about 7 too high, largest-first about 8 too low. Switching to 64-bit numbers, which carry more digits, does not cure this. It only hides it better: the two orders then disagree around the tenth decimal place instead.

Where you meet it in AI

This is not just a curiosity. The course comes back to it to explain why a chatbot can give different answers to the same prompt, with the same model and the same settings. Nothing random is happening. The server groups your request with other people's so it can process them together. The size of that group changes the order of a big sum, the score the model gives a candidate word shifts in the sixteenth decimal place, and two candidate words swap places.

Can it be fixed?

Partly. A method called Kahan summation keeps a running note of what each rounding threw away and feeds it back into the total. It works, but like every fix for this problem, it costs extra work.

Why it matters

Rounding error explains why two runs of the same code can disagree, why careful engineers think about the order of big sums, and why more digits make errors smaller without making them go away. How computers store numbers in the first place is its own concept in this series. When the same calculation gives slightly different answers, it is worth asking whether the order of the arithmetic changed.

Learn it step by step in Chapter 2 of our free course AI From Scratch: Where a Loss Function Comes From: Likelihood, Not Convention.