Mixed precision: faster AI training without losing the tiny updates

What mixed precision is

Computers can store a number with more or less detail. A 32-bit number keeps plenty of digits; a 16-bit number uses half as many bits, so it is rougher, but chips can crunch it much faster.

Mixed precision is a way of training an AI model that uses both. The heavy arithmetic, the model's guess (the forward pass) and the working out of corrections (the backward pass), is done with rough 16-bit numbers. Meanwhile, a full-detail 32-bit master copy of the model's settings is kept on the side, and that is where the corrections are actually applied.

What the video shows

The video lays out the trade-off. Rougher numbers make AI training up to about three times faster, but tiny updates get rounded away. Mixed precision gets the best of both: it does the heavy math roughly and keeps a precise master copy of the settings.

An everyday example

Imagine a savings notebook with room only for whole euros. You drop one cent into the jar, a thousand times. Each time, you write down the new balance rounded to the nearest euro, and each time it rounds straight back to the same number. After a thousand deposits there are ten more euros in the jar, and the notebook insists nothing has changed.

The fix is obvious once you see it: do your quick estimates in whole euros if you like, but keep the real balance in a ledger that records cents.

Why the tiny updates disappear

During training, each setting of the model is nudged by a very small amount at every step. The course runs the same kind of test on a model setting: start a setting at 1.0 and nudge it down by one millionth, a thousand times. The right answer is 0.999.

• With 32-bit numbers, the result lands very close to 0.999.
• With 16-bit numbers, the result is still exactly 1.0. Every single nudge was rounded away.

The reason is the gap between neighbouring numbers. Near 1.0, the closest 16-bit values are further apart than a typical update, so adding the update changes nothing. This happens with both common 16-bit formats, float16 and bfloat16. The second one can hold much bigger and smaller numbers, but it keeps even fewer digits of detail, so it does not escape the problem.

How much faster it is

In the course, switching the same models to 16-bit arithmetic on a laptop graphics card made training roughly two to three times faster, between 1.8 and 3.2 times depending on how hot the card already was. At the scale of the largest models, a speed-up like that is the difference between a training run of about five months and one of about two.

Why it matters

Mixed precision is the reason big training runs can use fast, rough arithmetic without quietly losing what the model learns. The 16-bit numbers bring the speed; the 32-bit master copy protects the tiny corrections. A related problem, very small corrections that turn into zero in 16-bit form before they even reach the settings, has its own fix, called loss scaling.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.