Documentation debt: why nobody can check what trained today's AI

What documentation debt is

Documentation debt is what piles up when an AI training set grows so large that nobody records what goes into it, and ends up too big to describe afterwards. The name comes from the researchers Bender and colleagues, who used it for datasets that were never documented and now cannot be.

It works like money debt. Skipping the paperwork saves time today, but the cost keeps growing, and past a certain size it can no longer be paid back. Nobody can go back and label billions of documents one by one.

What the video shows

The video makes one simple point: nobody can fully say what is inside the data that trained today's AI systems. The training sets are so huge and so thinly documented that no one can check them after the fact. Everything we do know about them exists only because someone took the time to look.

An everyday example

Imagine moving house and throwing everything into boxes with no labels. Writing "kitchen" or "papers" on each box as you pack takes a few seconds. Skip it, and a year later finding your passport means opening every box.

Now scale that up. A single month of Common Crawl, a free public archive of web pages that training sets usually start from, holds about 2.14 billion pages. At that size, "just open every box" stops being an option.

What looking has found

When researchers were able to inspect public training sets, they found things nobody had planned for:

• A filter with a hidden bias. In C4, a cleaned-up web dataset, a rule that dropped any page containing a word from a list of obscenities removed 42% of documents written in African-American English, against 6.2% of those in White-aligned English.
• Heavy repetition. One 61-word sentence showed up 61,036 times in that same dataset.
• Books of doubtful origin. About 12% of another public dataset, The Pile, was a books collection that came from a private torrent site. It was pulled offline in August 2023 after a copyright complaint.

Filtering, duplicates and data origins are each a topic of their own. What they share is how we learned about them: someone went and checked. The same is true of what we know about the people paid to label harmful text, and about the electricity a training run uses.

How to spot it

When you read about a new AI model, ask three plain questions:

• Is the training data published, or at least described in detail?
• Could an outside researcher inspect it for bias, repeats or copied material?
• Does anyone know where each part of it came from?

If the honest answer is no, its training data carries documentation debt. Whatever its makers say about its data, nobody else can verify it.

Why it matters

For the data behind most of the models people actually use, nobody can do that checking. So some very ordinary questions have no answer. Is the data skewed against some groups? Did it include copyrighted books? Had the model already seen the test questions it is scored on? Each of those is a question about data, and without a record it can only be argued about, never settled.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.