Cross-validation: a fair test for an AI when data is scarce

What cross-validation is

To know whether an AI model is any good, you have to test it on examples it never saw while learning. Otherwise you are grading a student with the exact questions they practised on. The usual approach is to set aside a slice of your data as a final exam, often called a held-out set. That works when you have plenty of data. When you have very little, every example you lock away is one the model cannot learn from, and the exam itself ends up too small to trust.

Cross-validation is the way out of that trap. You pick a number, called k, split the data into k equal parts (called folds, hence the name k-fold cross-validation) and train the model k times. Each time, a different part sits out and acts as the exam while the model learns from all the others. At the end you average the k scores.

What the video shows

The video lays out the dilemma: testing an AI needs examples it has never seen, but a small dataset cannot spare any. Its answer is to take turns. The data is cut into slices, each slice gets to be the exam exactly once, and the scores from every round are averaged into one number.

An everyday example

Imagine a teacher with a bank of only 50 questions. If she teaches with 40 and keeps 10 for the exam, the grade depends heavily on which 10 she happened to pick: an easy handful flatters the class, a hard one punishes it.

Now imagine she splits the bank into five packs of 10 and runs five rounds. In each round the class studies four packs and is examined on the fifth, and every pack takes its turn as the exam. Each question is used both for learning and for testing, never both in the same round, and the average of five grades says far more than any single small exam could.

How it works

With k = 5, for example:

| Round | Learns from | Tested on | |---|---|---| | 1 | parts 2, 3, 4, 5 | part 1 | | 2 | parts 1, 3, 4, 5 | part 2 | | 3 | parts 1, 2, 4, 5 | part 3 | | 4 | parts 1, 2, 3, 5 | part 4 | | 5 | parts 1, 2, 3, 4 | part 5 |

In every round the model is tested only on examples it did not learn from in that round. The price is time: you train the model k times instead of once.

A common misconception

Cross-validation does not make the model better. The course lists it next to the tools that stop a model from memorizing, but its job is different: it gives you a trustworthy estimate of how the model will do on data it has never seen. Think of it as a better scale, not a better diet. And like any test, it can still be run carelessly; Chapter 4 of the course points to sources on the right and wrong ways to set it up.

Why it matters

A score is only useful if you can believe it. With little data, a single split can make a model look better or worse than it really is, purely through the luck of which examples landed in the exam. Taking turns spreads that luck across every example, so the number you report is one you can stand behind.

Learn it step by step in Chapter 6 of our free course AI From Scratch: Getting It to Train, and Getting It to Generalise.