Bootstrap: how to put error bars on any AI score

What the bootstrap is

Every score you measure on a test set is partly luck. Pick a different batch of test examples and you would get a slightly different number. The bootstrap is a simple way to find out how much that number could move. It needs no formula, only the test examples you already have: you build lots of pretend test sets out of them, score the model on each one, and watch how far the score wanders. That spread becomes an error bar, a range that says "the real score is probably somewhere in here".

What the video shows

The idea in the video fits in one sentence: resample your own test examples thousands of times and rescore each copy. The score wobbles from copy to copy, and the size of that wobble tells you how much to trust the original number. In the example, a score of 0.41 really meant anything from 0.30 to 0.52.

In the course, that 0.41 is the F1 score of a model that flags defective factory parts. F1 is a single number that balances catching real defects against raising false alarms.

An everyday example

Imagine you kept the receipts from your last 10 lunches and want to know what a typical lunch costs you. Write each price on a slip, drop the slips in a hat, draw one, note the price and put it back. Do that 10 times and average what you noted. Because every slip goes back, some lunches appear twice in a round and others not at all, so each round gives a slightly different average. Play a few thousand rounds. If nearly all the averages land between, say, 11 and 14, that is your error bar. You never needed a formula.

How it works

1. Take your test set, with the model's result on every example.
2. Build a new test set of the same size by picking examples at random with replacement: a picked example stays in the pool, so it can be picked again.
3. Recompute the score on that new set.
4. Repeat thousands of times (the course uses 10,000).
5. Line up all the scores and drop the lowest 2.5% and the highest 2.5%. The middle that remains is a 95% interval.

When to use it, and when not to

The bootstrap shines where no ready-made formula exists, which covers many of the scores people report for AI systems: F1, averages taken across several categories, pass@1 (the share of problems a model solves on its first attempt) and grades handed out by an automated judge.

It has one blind spot: tiny test sets. With only 20 examples and a model that got most of them right, a random redraw can easily contain nothing but correct answers, so the top of the range hits a perfect score. The redraws also move in steps of one example out of 20, so they cannot draw a range finer than that. For a plain right-or-wrong rate, the course recommends the Wilson interval instead: a formula that behaves well at any size and involves no randomness.

Why it matters

A bare number like 0.41 invites comparisons it cannot support. Imagine a rival model scores 0.45: the gap looks like progress, yet it is much smaller than the wobble of the first score alone, so on its own it proves little. (Scoring both models on the very same examples allows a sharper check, called a paired comparison.) Reporting 0.41 together with its range of 0.30 to 0.52 tells readers how far the result could shift with different test examples, and keeps you from celebrating noise.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.