#204Dieses Video und der Text sind auf Englisch.
Prompt testing: why twenty examples can't prove a prompt is better
One good answer proves nothing. Learn what a fair prompt test needs: known answers, several versions, enough cases and a margin of error you can trust.
Im Detail
What prompt testing is
Prompt testing means checking whether a new way of asking an AI (a new prompt) really works better than the old one, using evidence instead of gut feeling. Saying "this prompt is better" sounds like a claim about one answer. It is really a claim about how the prompt behaves across many different requests, and one nice reply cannot show you that. You have to run each version on a pile of questions whose correct answers you already know, and count.
What the video shows
The video makes one point: twenty test cases are not enough to separate a good prompt from a mediocre one. The fix is to test on many cases with known answers, look closely at the ones whose result changed, and let results decide instead of opinions.
The numbers behind it come from the course. A prompt that gets 17 of 20 cases right scores 85%. Another that gets 13 of 20 right scores 65%. That looks like a clear win. But once you add the margin of error, the first prompt's true accuracy plausibly lies anywhere from about 64% to 95%, and the second's from about 43% to 82%. Those two ranges overlap almost entirely, so twenty cases cannot tell you which prompt is actually better.
An everyday example
Imagine two basketball players at a tryout. One makes 17 of 20 free throws, the other makes 13 of 20. It is tempting to pick the first one on the spot. But twenty shots is a small sample: a good day, a bad day or a bit of luck can easily produce that gap. Ask each of them to shoot a few hundred times and the real difference, if there is one, shows up.
Prompts behave the same way. A handful of tries mostly measures luck.
How to test a prompt properly
The course builds a small testing bench, about fifty lines of code, that runs every version of a prompt over the same set of cases in parallel and reports each score with a margin of error. The recipe is short:
- Cases with known answers. Collect real requests and write down the correct answer for each one before you start.
- Several versions. Run every prompt you are considering over all of the same cases.
- A margin of error, not just a score. The course uses a standard formula called the Wilson interval, which gives the range where the true accuracy most likely sits.
- A case-by-case comparison. Because both versions answered the same questions, study the cases where they disagree. That is far sharper than comparing two headline scores.
The chapter uses 60 cases. That is still a modest number: enough to spot big effects, and honest enough to admit when a small one is invisible. The chapter's measurements come from a small open model, with about half a billion parameters (the adjustable internal values a model learns), running on one consumer graphics card, so anyone can rerun them for free. A commercial model can be changed by its provider without warning, which makes its results hard to check later.
Why it matters
The course points out that much of the prompt advice you read online was checked on even fewer examples than twenty. Without a proper test, teams end up arguing about wording based on a couple of screenshots. With one, the question "is this prompt better?" gets a number and an honest margin of error, and the argument is over. The chapter sums it up in one line: a prompt is measured, not debated.
Learn it step by step in Chapter 15 of our free course AI From Scratch: Prompt Engineering, Measured: What Changes the Output.
Auch auf
Mehr aus diesem Kapitel
#204
#205Prompt order: the same words, rearranged, can double AI accuracy
#206System prompt: why where you put AI instructions changes results
#207Prompt formatting: how quotes and tags quietly shift AI accuracy
#208Few-shot prompting: how many examples an AI prompt really needs
#209Prompt myths: what tips, pep talks and threats did in a real test
Text mit AI-Unterstützung aus Kapitel 15 unseres kostenlosen Kurses geschrieben.