FLOPs: the six-operation rule for sizing an AI training run

What FLOPs are

A FLOP is one floating-point operation: a single multiplication or addition between two numbers with decimals, the kind of numbers AI models are built from. Under the hood, training a language model is a gigantic pile of these tiny steps, so the field measures the size of a training run by counting them.

The count comes from one short formula: C ≈ 6ND.

• N is the number of parameters, the adjustable numbers (the "settings") inside the model.
• D is the number of tokens the model reads during training. A token is a small piece of text, such as a word or part of a word.
• C is the total compute, in FLOPs.

In words: about six arithmetic steps for every setting, for every piece of text read.

What the video shows

The video makes the same point in one breath: training an AI is measured in tiny arithmetic steps, and the rule of thumb is about six of them for every setting, for every piece of text. Then it checks the rule against a real counter, which agreed within 17%.

Where the six comes from

Each training step has two halves.

• The forward pass is the model making its guess. Every setting is used once per token, in one multiplication and one addition. That is 2 operations.
• The backward pass is the model working out how to correct itself. At each layer it computes two sets of corrections: one for the layer's inputs, so the correction signal can keep travelling back through the model, and one for the layer's own settings, so they can be updated. Each costs about as much as the forward pass. That is 4 more.

Two plus four makes six.

An everyday example

Imagine a painter quoting a job before buying a single tin: wall area times number of coats. The quote is not exact, but it is close enough to plan a budget. 6ND is that quote for AI training.

A toy version: imagine a small model with 1 million settings reading 1 million tokens. The estimate is 6 × 1,000,000 × 1,000,000, or 6 trillion operations. Real models are vastly bigger, which is why the totals get astronomical.

Does the rule hold up?

The course does not just trust the formula; it measures it. Using a built-in FLOP counter in PyTorch, a popular AI programming toolkit, on models spanning a range of ten thousand times in size:

• The backward-plus-forward work came out at exactly three times the forward work, at every size. The "2 then 4" logic is not an approximation; it is arithmetic.
• The measured total landed 3% to 17% above 6ND, once the embedding tables are counted in N (more on that below). The gap is the attention scores, the step where each token is compared with the others around it, which the formula leaves out on purpose. It grows when the text window is long compared with the model's width.

One catch: the two founding studies of this field count N differently. One leaves out the embedding tables (the lookup lists that turn each token into numbers); the other includes them. For a small model with a big vocabulary, that choice changes N by a factor of nine.

Why it matters

A training run can cost a fortune, so you want to know its size before you start. 6ND lets anyone estimate that from two numbers they already know. Turning FLOPs into hours of hardware and into money is the next step, and a story of its own.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.