Train, validation and test: why the final exam stays sealed

What the three splits are

Before training an AI model, you divide your examples into three separate piles:

• Training set: the examples the model learns from.
• Validation set: the examples you use to make choices, such as which version of the model to keep, how fast it should learn, or where to set its alarm line.
• Test set: the examples you lock away and use only once, at the very end, to measure how good the finished model really is.

In the course's factory example, 20,000 parts are split into 12,000 for training, 4,000 for validation and 4,000 for the final test.

What the video shows

The video lays out the rule in its simplest form: one pile to learn from, a second to make choices with, and a third kept locked away. Every time you peek at that last pile, your final score becomes a little less honest.

An everyday example

Think about school. Homework is where you learn. Mock exams are where you find out what is working and change how you study. The final exam stays sealed until the day itself. If a student gets hold of the final paper in advance and practises on it, their grade may go up, but it stops telling anyone what they actually know.

Why two piles are not enough

It is tempting to skip the validation set and make your choices on the test set directly. The problem is that as soon as you use a pile of examples to pick anything, whether a setting, a model or an alarm line, that pile has helped shape the result. Its score is no longer a fair, independent check.

The course measured this. It chose the model's alarm line using the validation set, then scored the model once on the untouched test set: an F1 score (a combined measure of alarms that were right and defects that were caught) of 0.4122. Had it chosen the line on the test set itself, it could have reported 0.4186. That higher figure is one nobody has the right to claim, because the test set helped produce it.

The gap here is tiny, 0.006, because it came from a single choice made once against a large validation set. It grows with every extra decision you make and every time the validation set gets smaller.

A common misconception

"If peeking were a problem, I would see it in my numbers." Not on a single run. In the same experiment, the chosen line scored 0.3902 on validation and 0.4122 on test, so the validation score was actually the lower one that time. The bias from peeking is systematic: it shows up across many decisions, not in any one comparison you can point at.

How to use the splits

• Decide the split before you start, and keep the test set out of sight.
• Make every choice, including the small ones, on the validation set.
• Score the test set once, at the end, and report that number even if it disappoints you.
• If you go back and change things after seeing the test score, that test set is no longer a clean final exam.

Why it matters

A model's final score is a promise about how it will behave on data it has never seen. That promise only holds if the test set really was unseen. Each peek spends a little of its honesty, and you cannot get it back.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.