Test set size: how many examples an honest AI score needs

What test set size is about

A test set is the batch of examples you keep aside to grade an AI model once it is trained. How big should it be? It depends on how precise your score needs to be. Every score measured on a limited number of examples comes with an error bar, a range the true score probably sits in, and the fewer examples you use, the wider that range gets. Choosing the test set size means choosing enough examples for the number you report to actually mean something.

What the video shows

The video makes its point with three numbers. A score from 20 tests could be 15 points off. You need about 200 tests to get within ±5 points, and about 4,900 to get within ±1 point. Underneath sits one rule that never changes: halving the error always costs four times the data.

An everyday example

Imagine two pairs of headphones in an online shop, both showing 85% positive reviews. The first pair has 20 reviews, the second has 200. On the first, every single review is worth 5 points of that score, so two or three buyers who happened to get a faulty unit can drag it down by 10 or 15 points. On the second, each review is worth half a point, and a few unlucky buyers move the score far less. The screen shows the same 85% for both, but only the second number is steady enough to lean on.

The numbers

For a model that scores around 85%, the course works out the error bar (the 95% range) at different test set sizes:

| Test examples | Error bar, in points | |---|---| | 20 | about ±15 | | about 200 | about ±5 | | about 1,230 | about ±2 | | about 4,900 | about ±1 | | about 19,600 | about ±0.5 |

Read it from right to left, because that is how you will use it: decide the precision you need, then look up what it costs. Each time you want the error bar cut in half, you need roughly four times as many examples. Going from ±2 to ±1 points is not a small upgrade. It quadruples the test set.

Three traps

• Count the examples behind the number you quote. The course's factory test set holds 4,000 parts, but only 74 of them are defective. The share of defects the model catches (its recall) rests on those 74 parts, not on all 4,000, so it carries an error bar of about ±11 points.
• Examples must be truly separate. Three photos of the same part are not three examples, and five questions written from one document are not five. When items come in groups, count the groups.
• Look at the test set once. Every extra version you try against the same test set gives luck another chance to hand you a fake winner.

Why it matters

Leaderboards and product decisions often hinge on small gaps. With 1,000 test examples, the error bar is still around ±2 points, so a gap of a few tenths of a point between two models tells you nothing on its own. The exception is when both models were graded on exactly the same examples, which changes the arithmetic and is what a paired comparison is for. Knowing the price of precision lets you size the test set before you start, instead of discovering afterwards that your result was noise.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.