Scaling laws: why ten times more compute buys a predictable gain

What scaling laws are

A scaling law is a rule of thumb that predicts how much better an AI language model gets when you give it more resources. There are three of them: the number of parameters (the adjustable numbers inside the model, its "settings"), the amount of training text, and the compute, meaning the total amount of arithmetic the computers do while training.

"Better" is measured with the model's loss: a score for how wrong its guesses about the next word are on text it has never seen. Lower is better.

In 2020, Kaplan and colleagues trained a whole grid of models of different sizes and noticed something remarkably regular. As any of the three resources grows, the loss falls along the same kind of smooth curve, called a power law: every time you multiply that resource by ten, the loss shrinks by the same fraction. The pattern held over a range of more than a millionfold.

What the video shows

The video makes one idea visible: give an AI ten times more computing and its error drops by a predictable amount. Do it again and it drops again, by a similar share. The returns are small, but they never stop. That regularity is what turned training a bigger AI into something closer to a purchase than a bet.

An everyday example

Imagine polishing an old, scratched table. The first hour removes a good share of the scratches. To remove the same share of what is left, you need ten hours. Then a hundred. The table never becomes perfect, and every round costs ten times more than the last, but you always know roughly what the next round will buy you.

Draw that on a chart where both axes jump in steps of ten (1, 10, 100, 1,000) and the points line up in a straight line. That is the straight line the video's hook talks about.

How big the gains really are

The numbers are humbling:

• The course reproduces the pattern on an ordinary laptop: ten times more compute took 19% off the loss.
• In Kaplan's fitted rules, ten times more parameters removes only about a sixth of the loss that remains.

That sounds disappointing, and in a way it is. The important part is not the size of the gain but its reliability. A power law with a small exponent (the number that sets how steep the curve is) says each tenfold step removes the same small fraction of whatever loss is left. Since there is less left each time, every step helps a little less than the one before, and the help never runs out.

Why it matters

A dependable rule changes the question a company asks. It is no longer "will a bigger model be better?" but "how much better, and for how much money?". Spending on computing power stopped looking like a gamble and started looking like an investment with a published exchange rate. That argument is what unlocked the enormous training budgets behind today's large models.

What scaling laws do not promise

• They are fitted, not derived. Researchers measured many training runs and drew the best line through the results. Nobody proved the rule from first principles.
• They predict the loss, not skills. A lower next-word score does not tell you which abilities a model will gain.
• The exact constants are not universal. They shift with choices like how the text is chopped into pieces.

The same 2020 paper also gave advice on how to split a budget between model size and data, and that advice was later corrected. That is a separate story, known as Chinchilla scaling.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.