Repeating training data: how often an AI can reread the same text

What repeating training data means

Language models learn by reading enormous amounts of text, counted in tokens, the small pieces of words a model reads. One complete pass through all the training text is called an epoch. The famous scaling laws, the formulas that predict how much a model's error drops as you give it more computing, quietly assume that every token the model sees is a fresh one. In other words, they assume the supply of text never ends.

Repeating training data is what happens when it does end: the model has to read the same text again, for a second, third or tenth epoch. The question is how much that costs.

What the video shows

The video starts from that hidden assumption: AI scaling expects endless fresh text. Then it shows what happens when the text runs out. Going through the same text four times is almost as good as reading new text. But after about 16 passes, extra computing teaches the model nothing.

An everyday example

Imagine studying for an exam with only one textbook. Reading it a second, third and fourth time still teaches you things you missed, almost as much as a new book would. By the sixteenth read you are turning pages and learning nothing new. The effort is real, but the benefit has run out.

What the research measured

A study by Muennighoff and colleagues tested exactly this situation. Its headline comparison used one model size, 8.7 billion parameters (adjustable settings), trained two ways:

• on 44 billion tokens of unique text, each read four times;
• on 178 billion tokens of unique text, each read once.

Both versions read roughly the same total amount. The one that repeated its text ended only 0.5% worse on its validation loss: the error score measured on text the model never trained on.

Two thresholds are worth remembering:

| Passes over the same text | What the extra passes buy | |---|---| | Up to about 4 | Almost as much as fresh text would | | Beyond about 16 | Nothing: the extra computing is wasted |

How to use it

If you have less text than your computing budget would like, reading it up to about four times over is a reasonable choice and costs very little. If a plan calls for dozens of passes over the same text, most of that computing will not improve the model, and the effort would be better spent finding more text.

A common misconception

It is tempting to believe that more computing always means a better model. Scaling laws seem to promise exactly that. But that promise is built on fresh text. Once a model is rereading the same material for the sixteenth time or more, extra computing stops improving it at all.

Why it matters

This result tests one of the quiet limits of scaling laws, which the course lists alongside others: they forecast an error score rather than real skills, and they are fitted to experiments rather than derived from theory. Knowing how far repetition can stretch a limited supply of text tells you when more computing still helps, and when it is simply wasted.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.