Paired comparison: the fair way to tell two AI models apart

What a paired comparison is

When you test two AI models, the tempting move is to put their two scores side by side, each with its own error bar (the range its true score probably sits in), and check whether the bars overlap. A paired comparison does something sharper whenever both models answered exactly the same test questions: it compares them question by question. Instead of asking how uncertain each score is, it asks how uncertain the difference between them is. That turns out to be a very different, and much more useful, question.

What the video shows

Two models are tested on the same 500 questions. Their error bars overlap, which by a popular rule of thumb means "no clear winner". But look at the questions one at a time: the two models give different results on only 31 of the 500. On the other 469, they are either both right or both wrong. And of those 31 disagreements, one model wins 27.

An everyday example

Imagine two friends take the same 50-question pub quiz. On 45 questions they either both get it right or both miss it. Those questions say nothing about who is the stronger player: an easy question helps both, a brutal one sinks both. The whole contest lives in the five questions where they split. If one friend wins nearly all of those, that is the real evidence. Each total, on its own, also swings with how easy or hard this particular quiz happened to be, but that swing hits both friends alike and drops out once you compare them question by question.

Why pairing is sharper

In the course's numbers, model A scores 85.8% and model B scores 81.2%. Taken one at a time, A's error bar runs from about 82.6% to 88.8%, and B's from about 77.8% to 84.6%. They overlap, so the rule of thumb shrugs.

Now measure the difference item by item instead. A is ahead by 4.6 points, and the error bar on that difference runs from +2.6 to +6.8 points. The whole range sits above zero, so A really is the better model on this kind of test.

Why the change? Every test set mixes easy and hard questions, and that mix makes each separate score wobble. When both models face the very same questions, the easy and hard ones push both scores the same way and cancel out in the difference. What remains is the part that genuinely separates the two models.

A common misconception

"If the error bars overlap, the difference is not real." That rule of thumb looks at each score on its own and ignores that both models faced the same questions. On a shared test set it throws away the most useful information, and it can call a real winner a tie.

How to use it

• Run both models on exactly the same examples.
• For each example, record whether each model got it right.
• Put the error bar on the per-question difference, not on each score separately.
• Look at the disagreements: they carry the signal, while the questions both models got right, or both got wrong, cancel out.

Why it matters

Choosing between two prompts, two model versions or two suppliers is an everyday decision when building with AI. Shrinking error bars the usual way is expensive, because halving one costs four times the data. Pairing gets you to the same firm conclusion from a fraction of the examples.

Learn it step by step in Chapter 4 of our free course AI From Scratch: Classification, Cross-Entropy, and How Not to Fool Yourself.