Prompt Engineering, Measured: What Changes the Output

Before the measurements, one fact that quietly explains half of what follows.

The model has no memory. Between two calls it retains nothing — not your last question, not its own last answer, not the file you attached, not the fact that you asked it twice already. Every call starts from an empty machine, and the only thing that machine knows is the sequence of tokens you just handed it.

What looks like memory in a chat interface is your client resending the whole conversation, every turn, from the beginning. The model reads all of it again from scratch, every time. Chapter 13 measured what that re-reading costs in a forward pass; Chapter 16 turns it into a line on an invoice. What matters here is the consequence for design: the prompt is not a message to a system that has state. It is the state.

That retires a family of confusions. "The model forgot what I told it" usually means it was never sent. "It ignored my earlier instruction" usually means the instruction fell out of the window when the history was truncated. "It behaved differently in production" usually means production assembles a different prompt from the one you tested. None of these is a model problem, and none is fixed by rewording anything.

The claim "this prompt is better" is a claim about a distribution, and you cannot see a distribution by looking at one output. What you need is boring: cases with known answers, N variants, and an interval.

The harness is fifty lines of TypeScript with the same shape as the client from Chapter 14 — a request, a deadline, some concurrency, a tally. It reappears in Chapter 19 to evaluate a retriever and in Chapter 29 as the golden set.

export type Case = { input: string; expected: string };
export type Variant = { name: string; build: (c: Case) => ChatMessage[] };

async function pooled<T, R>(xs: T[], n: number, f: (x: T) => Promise<R>) {
  const out: R[] = new Array(xs.length);
  let i = 0;
  await Promise.all(
    Array.from({ length: n }, async () => {
      while (i < xs.length) {
        const k = i++;
        out[k] = await f(xs[k]);
      }
    }),
  );
  return out;
}

export async function runVariant(v: Variant, cases: Case[], concurrency = 6) {
  const hits = await pooled(cases, concurrency, async (c) => {
    const answer = await complete(v.build(c));      
    return answer.trim().toLowerCase() === c.expected;
  });
  return { name: v.name, hits, k: hits.filter(Boolean).length, n: cases.length };
}

The number that comes back is not the result. This is:

/** 95 % Wilson score interval for a proportion. Chapter 4 derives it. */
export function wilson(k: number, n: number, z = 1.96) {
  const p = k / n;
  const d = 1 + (z * z) / n;
  const centre = (p + (z * z) / (2 * n)) / d;
  const half = (z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n))) / d;
  return [Math.max(0, centre - half), Math.min(1, centre + half)] as const;
}

Chapter 4 made the argument and this chapter cashes it. Seventeen right out of twenty is 85 %, and its 95 % interval runs from 64 % to 95 %. A variant scoring 13 out of 20 — 65 %, which feels clearly worse — has an interval from 43 % to 82 %. Those two intervals overlap across almost their whole length. Twenty cases cannot tell a good prompt from a mediocre one, and most published prompt advice was validated on fewer.

Sixty cases, which is what this chapter uses, is still not many. It is enough to see large effects and honest enough to admit when it cannot see small ones — and it will admit that several times below.

Three blocks — the rules R, the ticket T, the instruction I — concatenated into one user message. All six permutations, byte-identical content, sixty cases each.

Best to worst is 28.3 points, and the intervals do not overlap, so this one is not a story about noise. Since every arm is scored on the same sixty items, the sharper question is the paired one: of the cases where two arms disagree, how lopsided is the split? Going from the worst order to the best flipped 21 cases right and 4 wrong — exact paired probability 0.0009.³

Read the table for its shape rather than its winner. The two best rows both end with the ticket; the two worst both bury the instruction in the middle or trail it after the data. That is the same phenomenon Liu et al. named Lost in the Middle: material at the edges of a prompt is used more reliably than material in the centre.⁴ Chapter 16 prices the window and Chapter 24 measures the effect properly at length, where the middle collapses as described and the recovery at the very end does not reappear. Here the practical rule falls out on its own: task at the top, data at the bottom, nothing important in the middle.

Now move the same words between turns. Chapter 11 established that the chat template is not decoration around the model but part of it — <|im_start|>system and <|im_start|>user are real tokens the model saw millions of times during fine-tuning, in exactly those positions. So it should matter which side of those markers your instruction lands on, and it does:

Moving the rules and the instruction across the template boundary bought 21.7 points — 19 cases gained, 6 lost, paired probability 0.0146 — without changing a character of them. This is the concrete answer to system prompt versus user prompt: they are not two ways of saying the same thing. They are two different token positions in a structure the model was trained on, and the system position is where instructions that apply to the whole conversation belong.

Notice the third row too. Repeating the instruction after the ticket — a widely recommended trick — scored below stating it once. On this model, on this task, saying it twice was worse than saying it once.

Same prompt, best placement, sixty cases. The only thing that changes is what surrounds the ticket text.

An eighteen-point spread from punctuation. But look at the two extreme intervals: [73.9, 91.9] and [54.1, 77.3]. They overlap. By the crude reading — compare the error bars, and if they touch, say nothing — this table proves nothing at all.

The crude reading is wrong here, and understanding why is worth more than the table. Every variant was scored on the same sixty tickets, so the two measurements are not independent samples; they are paired. Most of each interval's width comes from a source of uncertainty both arms share — whether these sixty tickets are representative — and that source cancels when you compare them against each other. Ask the paired question instead and the answer is sharp: going from double quotes to the XML tag flipped 12 cases right and 1 wrong, paired probability 0.0034. That is a real difference.

And then the same test deflates the headline. The XML tag beat the plain Ticket: label by 8.3 points, which is the number a blog post would put in its title. Paired: 6 gained, 1 lost, probability 0.1250. Not established. Seven cases is what that famous improvement rests on.

So there are two questions with two different instruments, and conflating them is how prompt advice goes wrong in both directions at once:

The general finding — that models are strongly and unpredictably sensitive to formatting choices carrying no semantic content — is not new. Sclar et al. varied nothing but separators, spacing and casing across dozens of tasks and found accuracy spreads wide enough to reverse published model rankings.⁵ The practical consequence is not "use XML tags". It is that formatting is a hyperparameter, it costs nothing to sweep, and any comparison of two models that fixes one format is comparing formats as much as models.

In-context learning — showing the model worked examples in the prompt and having it generalise from them without any weight update — is the capability that made GPT-3 famous.⁶ The practical question is never whether it works. It is how many examples to pay for.

Examples go in as real prior turns, alternating user and assistant, because that is the structure the template was trained on. Each k was run with five different random draws from a disjoint pool of sixteen labelled tickets:

Two examples bought seven points. The next six examples bought nothing measurable — 83.7, then 81.7, then 83.7, a sequence that wanders inside its own noise. Sixteen bought another five and a half. The curve is not a smooth climb; it is a step, a plateau and a step.

The column that matters most is the last one. At k = 8, which eight examples you happened to pick moved accuracy by 10 points — larger than the entire gain from going from two examples to eight. And the bottom row is the sharpest version of it: at k = 16 the pool is exhausted, so all five runs contain exactly the same sixteen examples, differing only in the order they appear. Order alone moved accuracy 8.3 points.

That is the result Lu et al. reported and it survives everywhere it has been looked for: example ordering is a genuine hyperparameter with effects comparable to example count.⁷ So the honest advice about few-shot prompting is not a number. It is:

Now the folklore. Each of these is a single sentence prepended to a system prompt that is otherwise identical, on the same sixty cases.

Four of the five did nothing. Not "did a little"; nothing that sixty paired cases can see. The expert persona and the bribe both scored below the untouched baseline, and even those drops fail the paired test — they are noise pointing downhill.

The third row is the one to sit with. "This is very important to my career" produced exactly the same accuracy, 46 out of 60 — and ten of the sixty answers changed, five in each direction. The summary statistic was identical and the behaviour was not. If your evaluation is a single number over a small set, a change that rewrites a sixth of your outputs can look like a change that did nothing, and you will ship it believing it was free.

And then the threat, which is the only sentence that moved the needle and moved it 35 points down, flipping 24 cases from right to wrong. That is not a rounding artefact; it is a different model behaviour. The lesson is not "never threaten a model". It is that emotional framing is not inert. It shifts the distribution, sometimes hard, in a direction nobody can predict from reading the sentence — which is precisely why it has to be measured rather than reasoned about.

A caveat this chapter owes you: these five sentences were tested on one small model and one task. Some have published support elsewhere — "take a deep breath" came out of a paper that searched for high-scoring instructions rather than inventing them, which is a different and better claim than the one that circulated afterwards.⁸ What generalises is not the sentences. It is that the list which survived in blog posts and the list which survives measurement are two different lists, and the only way to know which one you are holding is to run the bench.

A rule everyone repeats — say what you want, not what you do not want — with the usual absence of a number. Here is the number. The same format requirement, written three ways, with the model generating freely so that compliance can be observed:

Three prohibitions did worse than one instruction, and made the model write five times more text — the exact opposite of all three of them at once. Adding the positive sentence back rescued it to 68 %.

The mechanism is not mysterious once you remember Chapter 8. The model chooses a next token from a distribution conditioned on everything before it, and a prohibition puts the forbidden thing into that conditioning. There is no operator for negation; there is a context in which a word now appears.

Which is measurable directly. Take the baseline prompt and add one line: Do not use the shipping queue for software problems. Then look only at the forty-five tickets that are not shipping tickets:

Naming a queue in order to rule it out made the model choose it three times more often, nearly tripled the probability mass it assigned to it, and cost 25 points of overall accuracy — 16 cases lost against 1 gained, paired probability 0.0003.

Do not think of an elephant, measured. The rewrite is always the same: replace the prohibition with the positive rule that makes it unnecessary. Not "do not use shipping for software problems" but "use shipping only when a physical parcel is involved".

Chapter 12 built chain of thought properly — as a prompting technique first,⁹¹⁰ then as something trained in with verifiable rewards — and ended with a warning it deferred to this chapter: telling a model to think step by step stops helping once the model reasons on its own, and can hurt. Here is that warning with a table under it, on a task where it is easy to assume more thinking must be better.

Both arms are read with the same instrument at the same position. The only difference is whether a chain of thought the model wrote itself sits in the context first.

Accuracy went down and cost went up, and this chapter's own rule applies to this chapter's own result: the drop is 7 cases gained against 10 lost, paired probability 0.629, which is not established. What is established is that it produced ninety-eight extra output tokens per call and bought nothing measurable with them. The uncertainty is entirely on the benefit side. The bill is certain.

A chain that fails is more instructive than one that works. Asked to reason about "Your Slack integration stopped posting messages after Tuesday", the model wrote:

1. Check if the issue persists on Monday.
2. Verify if there are any updates or changes in your Slack setup that
   might affect message posting.
3. If no update has been made since Tuesday, check for any recent system
   restarts or downtime affecting Slack functionality.
4. If you have recently installed new software or updated your
   environment, ensure it's compatible with Slack version.
5. Contact Slack support for further assistance or troubleshooting steps.

That is competent troubleshooting advice and it is not the task. Asked to think, the model drifted into the genre that "think step by step about this support ticket" most resembles in its training data — and then answered a classification question with five hundred characters of unrelated reasoning in its own context. Chain of thought helps on problems with intermediate state worth computing: arithmetic, multi-hop lookups, constraint satisfaction. Routing a sentence into one of four buckets has no intermediate state. There is nothing for the chain to hold, so all it does is add plausible text that the final decision then has to survive.

Two practical corollaries. First, for a model trained to reason — the RLVR models of Chapter 12 — the instruction is worse than redundant: it can replace the long chain the model would have produced with a short, prompt-shaped one. And sampling several chains and voting, which is what self-consistency does,¹¹ cannot rescue a task with nothing to disagree about: it multiplies the cost by the number of samples to break ties that are not there. Chapter 12 measured that trade where it does apply. Second, note what the comparison scaffold itself cost. Forcing the answer into a Final queue: line dropped the no-reasoning arm from 76.7 % to 61.7 %. Fifteen points, paid to make the two arms comparable. Structure that exists for your convenience is not free either.

One last measurement, because it is the question everybody asks after the first surprising result. Sixty prompts, greedy decoding, run repeatedly:

• The same call repeated with everything held fixed returned bit-identical probabilities. Deterministic.
• The same call batched with different neighbours — batch sizes 1, 4, 12, 30 and 60 — returned probabilities differing by up to 0.0128. The chosen label never changed, in 0 of 60 cases.

The label survived because it had room to: across the sixty cases the narrowest gap between the top two queues was 0.0459, three and a half times the drift. The stability was not a property of the algorithm. It was a margin, and margins run out. Chapter 17 is where the arithmetic reason lives and where the sampling knobs that widen and narrow those gaps get taken apart. The reason to plant it here is that it bounds what any prompt measurement can mean: the bench measures a system reproducible only up to a tolerance, and a two-point difference between variants is inside that tolerance on a bad day.

Everything above is a human choosing a variant and a machine grading it. The obvious next step is to let the machine choose the variants too.

APE does exactly that: a model proposes candidate instructions, they are scored on held-out examples, and the best survive.⁸ The instructions it finds are frequently ones no human would write, which is the point — the search is over what scores, not over what sounds professional.

DSPy goes further and is the more useful idea for a product.¹² You declare what each step of a pipeline takes and returns, and the framework compiles that into prompts, selecting demonstrations and optimising instructions against your metric. Change model and you recompile instead of rewriting. The prompt stops being source code somebody hand-tunes and becomes an artefact generated against a metric, which is what it should have been all along.

Neither removes the need for the bench. Both make it the only thing you need, because an optimiser without a metric optimises nothing.

Which leaves the discipline. Prompts belong in version control, in files, next to the code that sends them — not in a database row somebody edited on a Tuesday. They need a version identifier stored alongside every output they produced, or the day something regresses you cannot find out what changed. They need the bench in continuous integration, because a prompt is the one part of your system that a vendor can silently invalidate by deploying a new model. And they need cases: not a hundred clever ones, just the boring twenty that broke last quarter, kept forever. The bench is the deliverable. The prompt is a by-product of it.

Everything in this chapter was measured in accuracy. Every one of those variants also has a price.

The system prompt that bought 21.7 points is sent on every call, forever. The two examples that bought seven points are sent on every call, forever. The sixteen that bought twelve are sent on every call, forever, and they are roughly ten times the length of the question the user actually asked. The chain of thought that bought nothing produced ninety-eight extra tokens per request, and output tokens are the expensive kind.

None of that is visible in a table of accuracies, and all of it is visible on an invoice.

Chapter 16 is about the unit those decisions are actually denominated in. The token as a billing unit, the context window as a budget rather than a memory, why a forty-turn conversation costs far more than forty times the first turn, what prompt caching does and does not pay for, and why the order of your prompt decides whether the cache hits at all — which turns out to be a second, entirely economic reason to put the stable material first and the variable material last.

The bench and every table were produced with Qwen/Qwen2.5-0.5B-Instruct under greedy decoding, so they reproduce exactly. The Hugging Face documentation on chat templates is the reference for what the template markers of Chapter 11 actually expand to, and for the fact that a model shipping the wrong template is a real and recurring failure. For the position and format effects at production scale rather than laboratory scale, the citations above are the primary sources; the vendor prompting guides are useful for their examples and should be read knowing that none of them publishes an interval.