コンテンツへスキップ

この動画とテキストは英語です。

Auto prompt engineering: letting the AI search for its own wording

AI Concepts #211

Instead of guessing the perfect wording, a model can write many instructions and keep the ones that score best on test cases. Here is how it works, and its limits.

詳しく

What automatic prompt engineering is

Prompts are normally written by hand, the way people write emails: someone picks the words that sound right, and maybe a colleague suggests a tweak. Automatic prompt engineering, often shortened to APE, hands that job to the machine. An AI model writes many candidate instructions for a task. Each candidate is tried on a set of examples with known right answers, examples kept aside so they played no part in writing the candidates. The instructions that score best survive, and the rest are thrown away. Nobody argues about wording: the scoreboard decides.

What the video shows

The video contrasts two ways of working. The usual one is guessing the perfect wording by hand. The alternative is letting an AI propose lots of instructions and test every one of them on cases where the answer is already known. And the twist: the winning instructions are often ones no person would ever write.

Why the winners look strange

A person writing a prompt aims for something that sounds clear, polite and professional. The search does not care how a sentence sounds. It only cares whether that sentence gets more answers right. So it will keep phrasing that reads as clumsy or odd, as long as it scores. That is the whole point of the method, not a flaw in it: what reads well to a human and what works for a model are not always the same thing.

An everyday example

Imagine a bakery choosing a name for a new loaf. The team could spend an afternoon debating names in a meeting. Or they could print twenty labels, try each one on the shelf, and keep whichever sells best. The winner might be a name nobody in the meeting would have picked, and that is fine, because the goal was sales, not applause in the meeting. APE works the same way, with instructions instead of labels and a test score instead of sales.

The famous example, told honestly

You may have seen the advice to tell a chatbot to "take a deep breath". It spread as a magic phrase. Its origin is more modest: it came out of this kind of automated search, run on one task with one model, where it scored well. That is a real finding, but a much narrower one than "this sentence makes every AI smarter". When the course tested it on its own small model and task, it made no measurable difference: 47 right out of 60, against 46 without it. A phrase that won someone else's search is a lead to test, not a rule.

What you need before you start

The search is only as good as its scoreboard. You need:

  • a clear task, with some examples of it;
  • a separate set of test cases with known answers, kept apart from anything used to write the candidates;
  • a score, such as the share of answers that come out right.

Without a score, there is nothing to search for. That is why this method does not replace careful testing; it depends on it completely.

Why it matters

APE turns prompt writing from a matter of taste into a matter of measurement. It also changes how you read advice online: any "best prompt" is best for a particular task, model and test set. Tools such as DSPy push the same idea further, from one instruction to a whole system.

Learn it step by step in Chapter 15 of our free course AI From Scratch: Prompt Engineering, Measured: What Changes the Output.

こちらでも

この章の他の動画

無料コース第15章をもとに、AIの支援を受けて作成したテキストです。