Model FLOPs utilization: how much chip speed AI training really uses

What model FLOPs utilization is

Every AI chip has a top speed on its spec sheet: the most arithmetic it can do in one second. That arithmetic is counted in FLOPs, short for floating-point operations, which simply means additions and multiplications on decimal numbers. Model FLOPs utilization, or MFU, is the share of that top speed that a real training run actually turns into useful work for the model. It is a percentage, and in real runs it sits well below 100%.

What the video shows

The video sets the chip's paper speed against what real AI training gets out of it: only part of the peak. One huge training run reported about 40%. And that share is what turns a compute estimate into hours, and hours into money.

The precise detail behind "about 40%": Llama 3's report gave a range of 38 to 43%, and the course rounds it to 40% for its cost calculations.

An everyday example

Imagine planning a 300 km road trip in a car whose speedometer goes up to 250 km/h. Nobody plans the trip at 250. There are traffic lights, junctions and a stop for fuel. If your real average is 100 km/h, the trip takes three hours, not 72 minutes. The top speed is true; it is just useless for planning.

MFU is that real average for an AI chip, written as a share of its top speed. In the car example it would be 40%.

How it works

Planning a training run starts with a rule of thumb from the same chapter: training takes about 6 × N × D operations, where N is the number of the model's adjustable settings (its parameters) and D is the number of tokens, the small pieces of words, it reads. That tells you how much arithmetic you need. It does not tell you how long it will take.

For that, you divide by the speed you really get:

| | Operations per second | |---|---| | One H100 GPU, rated peak | 989.5 trillion | | Share actually achieved (MFU) | 40% | | Useful speed per GPU | about 396 trillion |

At 40%, every job takes 2.5 times longer than the spec sheet suggests, and you pay for every one of those extra hours. Plan with the peak figure and both your schedule and your budget come out 2.5 times too optimistic.

Why the course keeps just this number

A big training run is split across thousands of chips in several different ways at once. The course deliberately leaves the engineering of that splitting to a dedicated university course, Stanford's CS336. What it keeps is this single number, because it is the one that turns an arithmetic estimate into hours on the clock, and therefore into money.

Why it matters

MFU is the bridge between an estimate on paper and a real bill. Two runs with the same model, the same text and the same chips can take different amounts of time if one keeps its chips busier than the other. When you read how much arithmetic a model took to train, MFU is the missing piece that tells you how many GPU-hours, and how much money, that really meant.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.