Inference-aware scaling: why labs train small AI models for longer

What inference-aware scaling is

Building a large language model, the kind of AI behind chatbots, comes with two very different bills. The first is training: the long stretch of computing that turns a mountain of text into a working model. You pay it once. The second is inference, which is simply the model answering people. You pay that every time anyone asks it anything, for as long as the model stays in service.

Inference-aware scaling means choosing the size of a model with both bills in mind, not just the first one. When you do the sums that way, the answer changes: the best choice is often a smaller model that was trained for much longer.

What the video shows

The video sums it up in one line: training is paid once, every answer is paid forever. Smaller models are cheaper each time they answer, so labs deliberately train them far longer than the famous "optimal" recipe would suggest, to get as much quality as possible out of a small size. That is why people say labs "overtrain" small models on purpose.

An everyday example

Imagine a delivery company buying a van. One van is cheap to buy but burns a lot of fuel. Another costs more up front but is far more efficient. If the van sits in the garage most of the year, the cheap one wins. If it drives all day, every day, for years, the efficient one wins easily, because fuel is the bill that never stops.

A popular AI model is the van that never parks. Extra training is the higher price up front. A smaller model is the lower fuel bill on every single trip.

How it works

A model's size is counted in parameters: the adjustable numbers it tuned during training. Each time a model writes one token (a word or a piece of a word), it does roughly two basic calculations per parameter. So a model with, say, 10 billion parameters spends around 20 billion calculations on every token it produces. Halve the size and you roughly halve that cost, on every token, for every user.

The well-known compute-optimal recipe (known as the Chinchilla result) answers a narrower question: for a fixed training budget, which mix of model size and training text gives the best model? It ignores everything that happens after launch. Researchers have since asked the bigger question:

• The authors of Meta's first LLaMA model argued that, for a target level of quality, the model you want is the one that answers fastest, not the one that trains fastest.
• Sardana and colleagues added the cost of answering to the cost of training and minimised the total. Their conclusion: if you expect something like a billion requests, pick a model that is smaller and trained longer than the compute-optimal recipe says.
• Llama 3's small models were trained well past the compute-optimal point, on purpose, swapping extra training work for cheaper answers.

A common misconception

"Overtrained" sounds like a mistake. Here it is not. It does not mean the model is broken or has memorised its training data. It only means it read more text than a training-only calculation would choose, because someone planned for the cost of serving it.

Why it matters

The compute-optimal recipe is not wrong. It simply solves a different problem from the one labs face once a model is popular. When a model will answer huge numbers of requests, the cheapest model to build is rarely the cheapest model to own. It also explains why a small model can be more capable than its size suggests: it may have been trained far longer than the textbook recipe would call for.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.