Chinchilla scaling: why a smaller AI that reads more can win

What Chinchilla scaling is

Training a language model has a budget: the total amount of computing you can afford. You can spend it on a bigger model, with more adjustable settings (called parameters), or on more text for it to read, measured in tokens (small pieces of words). Chinchilla scaling is the finding that, to get the best model out of a fixed budget, size and training text should grow together, in equal proportion.

It takes its name from Chinchilla, a model that Hoffmann and colleagues trained in 2022 to test the point.

What the video shows

The video starts with the old habit: for years, AI labs built giant models that read relatively little. Then it shows the correction. Size and text should grow together, and a model four times smaller, trained on about four and a half times more text, won on the same budget.

An everyday example

Imagine one fixed amount of money that must cover both a car's engine and its fuel. A huge engine with a nearly empty tank will not get you far; a medium engine with a full tank goes much further for the same price. Chinchilla showed that the big language models of the time were huge engines running on too little fuel.

The head-to-head

Earlier advice, from a study by Kaplan and colleagues, said to grow models much faster than their data. Models such as GPT-3 and Gopher were trained on somewhere between half a token and two tokens of text per parameter.

Hoffmann's team trained more than 400 models, ranging in size from 70 million up to 16 billion parameters, and three separate methods all pointed the other way. Then came the direct test:

| | Gopher | Chinchilla | |---|---|---| | Model size | 280 billion parameters | 70 billion parameters | | Text read | 300 billion tokens | 1.4 trillion tokens | | Computing budget | same | same | | Score on MMLU, a test spanning 57 subjects | 60% | 67.5% |

It also won on 51 of the 57 subjects. The rule behind it fits in one line: each time you double the model's size, double the training text too.

How it works in practice

The course repeats the method on a laptop, at a millionth of the scale. Pick a budget, train many model sizes on exactly that budget, and compare the results. Two lessons appear straight away. A model can be too big for its budget just as easily as too small: in one case, picking a model a little more than twice the best size wasted close to a fifth of the budget. And as the budget grows, the best size moves up with it, at a rate close to the one the paper reported.

A common misconception

"Chinchilla says 20 tokens per parameter." The paper never states that ratio. It only says that model size and training text should double together; the 20 is read off the paper's numbers, partly from Chinchilla's own 70 billion parameters on 1.4 trillion tokens. A later re-analysis by Besiroglu and colleagues found the original estimates were less precise than published and put the plausible range anywhere between 4 and 40 tokens per parameter. The direction of the rule holds up; the exact ratio does not travel well. In the course's small laptop runs, the best models read between about 170 and 540 tokens per parameter.

Why it matters

Chinchilla's message was that the large language models of its day had been trained on far too little text for their size. Today many teams go further and train smaller models for even longer, because a smaller model is cheaper every time it answers; that is a separate trade-off built on top of this one.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.