Deduplication: why removing repeated text makes AI copy less

What deduplication is

The web repeats itself all the time. The same cookie notice, the same legal footer, the same product blurb and the same viral sentence turn up on page after page. When that text becomes training data for a language model, the repeats come along too.

Deduplication is the step that finds those copies and removes them, so a piece of text appears once, or at least far fewer times. It sounds like tidying up a folder. It turns out to change how the model behaves.

What the video shows

The video's hook is a single number: one sentence showed up 61,036 times in a single AI training dataset. Training data is full of repeats like this, and models memorise them. Removing the copies cut memorised output tenfold. Then comes the twist: removing too much brought no gain at all.

An everyday example

Imagine revising for an exam from a big folder of notes in which one page has been photocopied sixty thousand times and shuffled in with everything else. You would end up able to recite that page word for word, not because it matters most, but because you saw it most. A model trained on heavily repeated text does something similar: it learns to reproduce that text exactly, instead of learning the general patterns we actually want.

What the numbers say

• The repeats can be extreme. Lee and colleagues found a single 61-word sentence that appeared 61,036 times in C4, a large cleaned-up web dataset.
• Removing them cuts copying. Memorised text means passages the model repeats word for word from its training data. After deduplication, that share fell from 1.9% of the tokens the model generated to 0.19% (a token is a word or a piece of a word). That is roughly one in fifty down to one in five hundred.
• More is not automatically better. The team behind the FineWeb dataset first removed duplicates across all 96 of their web crawls at once (a crawl is one sweep that collects a snapshot of the web). That left 4 trillion tokens and brought no measurable improvement. When they removed duplicates inside each crawl separately instead, they kept 20 trillion tokens and matched the best dataset available at the time.

How far to go

Deduplication is a normal step before training on web text. The open question is how hard to push it. Removing thousands of copies of the same long passage clearly helps. Removing everything that looks like a repeat, across the whole collection, can throw away a huge amount of text without making the model any better.

The FineWeb result is the practical reminder: measure the effect of each cleaning step on the finished model, rather than assuming that cleaner data is always better data.

A common misconception

It is tempting to think of deduplication as boring housekeeping, the data version of deleting duplicate photos from a phone. The evidence says otherwise. How many times a model sees the same words changes what it memorises, and how aggressively you remove copies changes how much useful text is left to learn from. Both are real design decisions.

Why it matters

We want a model to learn patterns, not to recite pages. A model that repeats its training text word for word is less useful, and it can repeat things that were never meant to be repeated. Deduplication is one of the cheapest ways to reduce that, as long as you remember that the goal is fewer copies, not less data.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.