Loss scaling: how AI training saves corrections too small to count

What loss scaling is

When an AI model trains, it makes a guess, measures how wrong it was, and works out a small correction for every one of its internal settings. Those corrections are called gradients: each one says which way to nudge a setting, and by how much. Some of them are astonishingly small, around 0.00000001.

That becomes a problem when training uses 16-bit numbers, a compact way of storing numbers that takes half the memory of the usual 32-bit format and lets the chips work faster. The common 16-bit format, called float16, cannot store a value that tiny. It rounds it to exactly zero, and the correction is gone. Loss scaling is the trick that rescues it: make every correction much bigger while the fragile 16-bit numbers are handling them, then shrink them back once they are safe in a more exact format.

What the video shows

The video shows the problem first: some training corrections are so small that fast 16-bit arithmetic turns them into zero. Then it shows the fix. Loss scaling enlarges all the corrections at once, so they land in a range 16-bit numbers can hold, and later shrinks them back to their true size using exact, full-precision numbers.

An everyday example

Imagine trying to measure the thickness of a hair with a school ruler. The smallest mark is a millimetre, so the ruler says zero. Now photograph the hair through a microscope that magnifies it 1,000 times. On the photo the hair is easy to measure, and you divide the result by 1,000 to get its real thickness.

Loss scaling does the same thing with numbers. The microscope is a multiplication, and the division at the end gives you the true value back.

How it works

1. The model scores how wrong its guesses were. That score is called the loss.
2. Before any corrections are worked out, training multiplies the loss by a large fixed number. The course's example uses 65,536.
3. The corrections are then calculated backwards from the loss, a step called backpropagation. Every correction is built from the loss through a chain of multiplications, so every single one comes out 65,536 times bigger. None is favoured; they all grow by the same factor.
4. A correction of 0.00000001, which float16 would have erased, now arrives as roughly 0.00066, a size float16 can store.
5. The corrections are moved into full-precision 32-bit numbers and divided by 65,536 again. In the course's test, the tiny correction comes back as 0.000000009997, almost exactly the original.
6. Only then is the model updated.

What it does not change

It is tempting to think that multiplying the loss changes what the model learns. It does not. The enlargement is undone before any setting is changed, so each correction goes back to its true size, give or take a tiny rounding error; the only difference is that the smallest ones are no longer turned into zero on the way. The multiplication just protects the numbers while they pass through the fragile format.

And not every 16-bit format needs it. A variant called bfloat16 keeps fewer digits of detail but covers the same range of sizes as 32-bit numbers, so the 0.00000001 correction survives there and loss scaling is mostly unnecessary.

Why it matters

This is not a rare corner case. A study of a real network found that roughly 5% of its weight corrections were small enough to vanish in float16. Nothing crashes when that happens and no warning appears; part of the learning signal just quietly disappears. Loss scaling is a cheap fix that lets people keep the speed of 16-bit training, the idea behind mixed precision, without throwing that signal away.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.