Data filtering: the blunt rules that decide what an AI reads

What data filtering is

A large language model learns from text, and most of that text comes from the public web. The raw pile is enormous: a single month of Common Crawl, a free public archive of the web, holds about 2 billion pages. Most of it is not worth learning from. Think menus, error messages, copied boilerplate and plain gibberish.

Data filtering is the step that decides which of those pages the model gets to read and which get thrown away. It sounds like housekeeping. In practice it is one of the biggest decisions in building a model, because a model can only learn from what survives the filter.

What the video shows

The video starts from that pile of billions of web pages, mostly junk, and shows the simple filters used to clean it up. Then it shows the problem: an audit found that one of those filters, a list of banned words, removed texts written in some dialects of English far more often than others.

How the filters work

The best-known example is C4, a cleaned-up web dataset introduced alongside a model called T5. Its rules are deliberately simple:

• keep a line only if it ends with a punctuation mark, such as a full stop;
• throw out any page shorter than three sentences;
• throw out any page with a curly brace, the { symbol that usually signals computer code;
• throw out any page that uses a word from a published list of obscene words.

Rules like these shrank about 20 terabytes of text from one month of crawling to about 750 gigabytes. Put simply, roughly 96% of it was thrown away.

Where it went wrong

Dodge and colleagues checked what those rules actually removed. The obscene-word list threw out 42% of the texts written in African-American English. For Hispanic-aligned English the figure was 32%. For White-aligned English it was only 6.2%.

Nobody wrote a rule about dialect. The list only named words. Yet its effect fell far harder on some ways of writing English than on others, so the model ended up learning from a narrower slice of how people actually write.

An everyday example

Imagine a school email system that blocks any message containing a word from a banned list. It never looks at who sent the message. But if a few of those words are ordinary in some students' everyday writing, their messages get blocked far more often, and the school ends up hearing much less from them. The filter looks neutral. Its effect is not.

Another way filtering can fail

Filters can also let the wrong things in. Benchmarks, the standard tests used to score AI models, are published online, so their questions can slip into the training data. This is called contamination: the model may have seen the exam before sitting it. Meta measured this for Llama 3 and found heavy overlap with several popular tests. The GPT-3 team reported a filtering bug that left test material in their data, and retraining to fix it was too expensive.

How to spot the problem

• Look at what a filter removes, not only what it keeps, and break the removals down by group.
• Read a random sample of rejected pages yourself.
• Check the training data for overlap with any test you plan to report.

Why it matters

A filter is a decision about what "good text" looks like, taken at a scale no person could review. Blunt rules are cheap and fast, which is why they are popular. But they shape the model's picture of language, so a rule that seems neutral deserves the same scrutiny as any other design choice.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.