LLM Evaluation: From Public Benchmarks to Your Golden Set

Almost every argument about evaluation is two people measuring different things. There are three projects and they share no instrument.

The confusion is expensive in one direction. A leaderboard tells you a model is strong at graduate-level reasoning; it cannot tell you whether it will route your support tickets. And an application evaluation that scores one answer per input cannot see an agent at all, because an agent has a distribution of trajectories and one answer is a single sample from it. Chapter 22 named that third row and left it empty: the performance measure, the one part of an agent's specification that teams write down last or never.

The order matters too, and the vendor selling you the model says so. OpenAI's agent guide reduces model selection to three steps, in this order: "Set up evals to establish a performance baseline", "Focus on meeting your accuracy target with the best models available", "Optimize for cost and latency by replacing larger models with smaller ones where possible".¹ Evaluation comes first, because steps two and three are meaningless without a number.

A golden set is a list of inputs, each with the answer written down, and a grader that decides whether an output matches. It is boring, it is small, and it is the only artefact in this chapter that is yours. The one built here has twenty tasks over a directory of five files — not Chapter 23's three, so the answers are not the same answers — and the grader is written before the agent runs:

export type Task = {
  id: string;
  prompt: string;
  answer: string;          // the fact, in words, for a human and for a judge
  must: RegExp[];          // ALL must match the final answer
  mustNot?: RegExp[];      // NONE may match
};

export const GOLDEN: Task[] = [
  { id: "t04", prompt: "Which file is the largest?", answer: "access.log",
    must: [/access\.log/i], mustNot: [/errors\.log/i, /notes\.txt/i] },       
  { id: "t12", prompt: "Which HTTP status codes appear in access.log? List all of them.",
    answer: "200, 429 and 500", must: [/200/, /429/, /500/] },                
  // ...eighteen more
];

Two properties are load-bearing. The mustNot list exists because a model that names three files including the right one has not answered. And answer is written in prose as well as in patterns, because a human and a judge will both need it later — and writing the same fact twice in two notations is how you find out you did not agree with yourself about what the task was.

Now the table that decides. Four candidate systems, the same twenty tasks, accuracy with its interval, and the two columns that a table of accuracies alone always hides:

Read the intervals before the winner. Arm B's runs from 11 % to 47 %; arm A's from 3 % to 30 %. They overlap across most of their length, which is Chapter 4's finding arriving exactly where it was promised: twenty cases cannot rank four systems. Chapter 15 sharpened this by asking the paired question instead — of the cases where two arms disagree, how lopsided is the split? — because the shared difficulty of the set cancels. Here is every pair:

A vs B  +0 / -3   p = 0.2500       B vs C  +4 / -1   p = 0.3750
A vs C  +2 / -2   p = 1.0000       B vs D  +4 / -0   p = 0.1250
A vs D  +2 / -1   p = 1.0000       C vs D  +1 / -0   p = 1.0000

Not one of the six comparisons is established. The best arm beats the arm with no tools at all by fifteen points, and three discordant cases is what that rests on. Twenty cases show a mechanism and cannot choose a supplier; saying otherwise in a meeting is how a bad model gets bought.

There is one thing this table does establish, and it is the column nobody puts in. Arm D costs thirteen times arm B per solved task, because sampling three trajectories and taking the modal answer triples the bill whether or not it triples the accuracy. Accuracy tables that omit cost make that trade invisible.

Now the finding that changes how you read every benchmark you will ever see. Take the same two hundred transcripts — twenty tasks, ten runs, not one token regenerated — and score them three ways:

Zero, thirteen, twenty-six. The system did not change. The grader did. Exact match returns zero not because the agent is useless but because no free-text answer is ever byte-identical to a reference: it measures formatting and reports it as capability.

That is not a curiosity, it is a mechanism, and it has a name. A hard-cutoff metric scores a task all-or-nothing over several sub-facts, so it compounds. Task t12 asks for three status codes at once. Over the ten runs:

per-code presence   200: 9/10    429: 6/10    500: 8/10    (mean 0.77 per fact)
all three at once   5/10

Each fact is right about three quarters of the time; demanding all three at once halves the score, and $0.77^3 = 0.457$ is close enough to the measured 0.50 to show where the drop came from. Generalise:

Read the 0.90 row against the 0.95 row at $k = 10$: a per-fact improvement of five points becomes twenty-five points on the conjunction. Nothing discontinuous happened to the model. A smooth curve read through an all-or-nothing metric looks like a jump — which is precisely the argument Schaeffer, Miranda and Koyejo made about emergent abilities, and which Chapter 10 deferred to here.² Their audit found that at most 5 of BIG-Bench's 39 preferred metrics display emergence at all, with two discontinuous metrics accounting for over 92 % of claimed cases.

So the discipline, in one line: a jump in a chart is evidence about the metric until shown otherwise. Before you believe a capability appeared, plot the same runs with a metric that gives partial credit and see whether the cliff survives.

There is a second-order version of this that Kalai and colleagues argue is doing damage upstream: benchmarks scored as right-or-wrong reward guessing over saying "I do not know", so a model optimised against them learns to guess. Their proposed fix is not another hallucination benchmark but "modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards".³ Your golden set has the same lever, and it is one line: decide whether an abstention counts as a failure or as its own category. Most people never decide, so it silently counts as a failure, and the system they ship guesses.

Everything so far scored one attempt per task. An agent is not one attempt. Chapter 17 established that you do not have determinism even at temperature zero, so the same input produces a distribution of trajectories and a benchmark that runs each task once reports one sample from it.

τ-bench's contribution is the metric for that. The paper defines it plainly: "we propose a new metric – pass^k (pass hat k), defined as the chance that all k i.i.d. task trials are successful, averaged across tasks."⁴ Run each task $n$ times, count the $c$ successes, and the unbiased estimators are:

The second one is the familiar pass@k from code generation: the chance that at least one of $k$ attempts succeeds. Put them side by side on the same measured counts and they move in opposite directions:

Same runs, same grader, same twenty tasks. One column says the system improves with more attempts and the other says it gets worse, and both are correct, because they answer different questions. pass@k is the right metric when a human filters the output — code generation, drafts, brainstorming — and the extra attempts are cheap. pass^k is the right metric when the agent acts without a filter, which is what "agent" means. Publishing the first where the second applies is the most common overstatement in this field, and τ-bench's own headline is the honest version: gpt-4o at roughly 61 % pass^1 on retail falls to about 25 % at pass^8.⁴

Now the sting in my own numbers. pass^10 over my twenty tasks is 5.0 %: exactly one task in twenty solved on all ten runs. That task is t19, "Did deploy 42 succeed?", and here are two of the ten answers the rubric scored as correct:

run 2  "To check if 'deploy.log' succeeded in deploying 42, I will list the file
        names in the working directory using the list_files function..."
run 8  "Yes, deploy 42 has successfully deployed. Deploying was successful for 41
        as well."

The first never answers. The second adds a claim that is false — deploy 41 was rolled back. Both matched /succe|yes/. The only task holding pass^10 above zero is a grader artefact, so the true figure is zero, and no aggregate would have shown me that. Sampling the transcripts behind your best-scoring task is where graders go to die.

And one more number, the one this section is named for. Ten identical evaluations — same system, same twenty tasks, same code, nothing changed but the seeds:

per-run correct: 5 2 5 5 8 5 8 5 4 5   ->  10 % .. 40 %,  mean 26.0 %,  sd 8.8 points

A thirty-point range on a system that did not change. If you run your suite once before a release and once after, an eight-point "improvement" is inside that spread and you will ship it believing you caused it. This is why the pooled interval above — 26.0 % [20.4, 32.5] — is too narrow to quote on its own: it treats two hundred correlated trials as two hundred independent ones. The honest summary of an agent evaluation is a mean and a spread across repeats, and almost nobody publishes the second.

Rubrics do not scale to open-ended answers, so the standard move is to have a model grade the output. It works well enough at frontier scale to be the default, and it has three named failure modes: position bias, verbosity bias and self-enhancement bias.⁵

Measure it before trusting it. The same sixty answers — three of the ten runs — were labelled three ways. The human label is mine: I read all sixty with the five files open and applied one written rule, pass if and only if the answer states the fact the question asked for and contains nothing contradicted by the files.

The judge said PASS sixty times out of sixty. It would have reported this agent at 100 % accuracy on a set where the human scores it at 18 %. A judge with no discriminative power is not a noisy instrument; it is a constant function, and a constant function gives your best system and your worst system the same score.

Prompting did not rescue it. Four variants, same sixty items:

Swapping the order of the two labels in the instruction moved four verdicts. That is a measurable effect and it is the wrong kind of effect: the judge is responding to the shape of the prompt rather than to the answer in front of it.

The clean demonstration is pairwise. Twenty questions, each with one plainly correct and one plainly wrong candidate, presented in both orders:

picked the FIRST option              40/40 = 100.0 %
order-consistent (same winner both ways)   0/20 = 0.0 %   [Wilson 0.0, 16.1]
picked the CORRECT answer            20/40 = 50.0 %

It picked position A forty times out of forty. The 50 % on correctness is not partial competence — it is arithmetic, because the correct answer sits in position A on exactly half the trials. Consistency here is defined as MT-Bench defines it, "the percentage of cases where a judge gives consistent results when swapping the order of two assistants", which lets the comparison be apples to apples: GPT-4 scores 65.0 % on that measure, and few-shot prompting lifted it to 77.5 %.⁵ Mine scores zero.

The standard mitigation is also from that paper: "call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders."⁵ Apply it here and the judge produces zero usable verdicts from twenty pairs — which is the correct outcome, and infinitely better than twenty confident ones.

A methodological note worth more than the result. I also ran a verbosity test: the same correct answer, one copy padded with a 36-word sentence that adds nothing. The judge preferred the longer version on exactly 50 % of trials — which looks like an absence of verbosity bias and is nothing of the sort, because a judge that always picks position A scores 50 % on any balanced pairing whatsoever. You cannot measure a second bias until the first is controlled. Swapping positions is not a refinement to add later; it is what makes every other measurement interpretable.

The honest caveat: this judge is a half-billion-parameter model, and nobody should grade with one. The point is not that judges are bad. It is that the numbers above cost eight minutes to produce, and without them this judge's verdict on a shipping decision would have been 100 %.

This is the third and last declared Python panel of the course, and the reason is where the public numbers come from. lm-evaluation-harness covers "over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented" and is "the backend for Hugging Face's popular Open LLM Leaderboard"; HELM, SWE-bench and τ-bench are Python packages with Python entry points.⁶ Running your model against a published figure means running their code, and the day you want to compare with a number somebody cited, this is the ecosystem you are in:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

The second reason is that one measurement in this chapter is impossible over HTTP. Contamination — the test set having leaked into the training data — is the failure that makes a public benchmark silently meaningless, and the sharpest probe for it needs the model's own loss, which no chat API returns. It is Chapter 8's cross-entropy per token, pointed at a question about memory:

def nll(text: str) -> float:
    """Mean negative log-likelihood per token, in nats."""
    ids = tok(text, return_tensors="pt").input_ids.to(model.device)
    with torch.no_grad():
        out = model(ids, labels=ids)
    return float(out.loss)

Ten sentence pairs: five in every crawl of the web since it existed, five written for this chapter this morning, each paired with a reworded version carrying the same content.

The model is four times more surprised by a sentence written this morning than by one it has seen a million times, and rewording costs twice as much on the famous ones — the extra cost being the part that was memorised rather than understood. Absolute loss confounds memorisation with ordinary naturalness, so the gap is the better statistic and the continuation test is better still. Give it the first six words:

famous  "Permission is hereby granted, free of"
     -> "charge, to any person obtaining a copy of this software and associated
         documentation files (the "
famous  "All human beings are born free"
     -> "and equal in dignity and rights. The right to life, liberty, and security"
fresh   "All evaluation harnesses are born tiny"
     -> ", and the most common way to measure their size is by using a ruler."

Three of the five famous strings continued word-perfect from six words; none of the five fresh ones did. That is a half-billion-parameter model reciting the MIT License. If your benchmark is on the public web, assume it is in the weights. It is also the argument for the whole chapter: a golden set you wrote from your own data, kept out of any repository a crawler reads, is the only test set you can be sure was never trained on.

They are still worth reading, as long as you read what each one measures rather than the single number attached to it.

Take the table rather than any row. The agentic benchmarks all put humans far above models, which is the opposite of the knowledge benchmarks and the best one-line summary of where the field is; their figures age within months, so cite them with the date you read them; and every one measures a task that is not yours.

Accuracy is the metric you argue about. These are the ones that decide whether the thing ships. All four fall out of the two hundred runs already measured.

Cost per solved task, not per call. The agent costs $0.001345 per attempt and $0.005172 per task actually solved — 3.85 times more, because three quarters of the attempts produce nothing. Latency behaves the same way: 1,213 ms per attempt, 4,667 ms per solved task. Every retry, every re-ask, every abandoned trajectory is on the second number and invisible in the first.

A diagnostic that beats accuracy. In 123 of 200 attempts the agent answered without calling a single tool — it guessed rather than looked. Splitting on that:

answered without reading anything   8/123  =  6.5 %  [3.3, 12.3]
answered after reading something   44/77   = 57.1 %  [46.0, 67.6]

The intervals do not come close to touching. That is worth more than the aggregate 26 %, because it names the thing to fix — the model is not failing to reason, it is failing to look — and the fix is in the harness, not the model. One caveat this chapter owes its own standards: the two groups are different tasks, not the same tasks paired, so part of that gap may be that it skips the tools precisely on the questions it finds hard. The split is a diagnostic, not a causal claim.

Human intervention rate is the metric a buyer asks for first: what fraction of runs stopped at an approval, a guardrail or a handoff. Chapter 23's typed interruptions make it countable, and counted per task type and per week it is what separates an agent learning its job from one quietly becoming a queue.

Abandonment is the one no offline suite can see: the user who read the answer, closed the tab and did the task themselves. Offline evaluation is a gate; production evaluation is a continuous sample of real traffic, scored on the same grader plus these four.

And a rule inherited from Chapter 17: never assert on exact output. Assert on properties — valid JSON, correct schema, the right tool called, a number within tolerance, a required substring present. The exact-match column at the top of this chapter is what happens when that rule is broken.

Evaluating a supplier is not only about accuracy, and this is the second half of this course's ethics, with its own heading rather than an appendix.

Measure the bias, do not assume it. Whatever you believe about a model's behaviour on names, dialects, genders or nationalities, it is a measurable property of your pipeline, and the instrument is the one you already have: take your golden set, vary only the attribute, compare paired. HELM exists precisely because accuracy alone was being reported where bias, toxicity, calibration and robustness were also decidable.⁸ A vendor's model card is a starting point, not evidence about your inputs.

Contamination is a supplier question too. The probe above is the reason to ask what a published number was measured on, and when the model's data was cut.

Retention, training and residency, read on 7 September 2026. These change, so record the date beside the answer. Anthropic's policy page states: "By default, we will not use your inputs or outputs from our commercial products (e.g. Claude for Work, Anthropic API, Claude Gov, etc.) to train our models", with the exception of content you explicitly submit as feedback, which is stored "for up to 5 years".¹⁶ OpenAI's data controls documentation states that "data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in to share data with us)", describes a thirty-day default retention for abuse-monitoring logs, and offers Zero Data Retention, which "excludes customer content from abuse monitoring logs", plus configurable data residency across a list of regions.¹⁷

Four questions to get in writing before the first production call, because each one has a different owner: is my data used for training; how long is it retained and by whom; where is it processed and stored; and what happens to all of that if I use a reseller, a gateway or an aggregator rather than the provider directly. That last one is where most surprises live, and no benchmark will tell you.

You now have the instrument: a golden set you own, an interval on every number, a paired test for every comparison, pass^k for the runs you did not show anyone, a measured judge, and a probe for whether a public score means anything. Chapter 23's closing claim can now be checked instead of asserted — a harness makes an agent governable, not correct — and checking it took two hundred runs and eight minutes.

There is one property of an agent that none of it measures, and it is the one that gets people fired.

Every task in this chapter's golden set was written by me, and every file the agent read was written by me. Nothing in that directory was trying to do anything. Change one line in one file the agent is told to read — a line that ends with an instruction addressed to whatever reads it next — and the agent that scored 26 % will follow it with the same tools, the same permissions and the same clean trace, and every number in this chapter will stay exactly where it is. An evaluation suite measures how often a system reaches your goal. It does not measure how easily somebody else can substitute theirs.

Chapter 30 is that: prompt injection, the lethal trifecta of private data, untrusted content and external communication, and what it costs to give an agent real permissions. It opens with the observation this chapter has been avoiding — that the same passing score is compatible with an agent that does exactly what an attacker wrote in a file it was told to read.

Every number above was produced on one machine and none of it touched a paid endpoint. The agent is Chapter 23's loop with two of its four tools over a five-file directory; the model behind the port is Qwen/Qwen2.5-0.5B-Instruct, exposed through a small server of the same shape as a chat completions endpoint exactly as in Chapter 23, but in half precision on one consumer GPU rather than that chapter's CPU. Costs use Chapter 16's rates — $2.00 per million input tokens and $12.00 per million output — applied to measured token counts. The repeated runs use temperature 0.7 with fixed seeds so the whole set reproduces; the four-arm table is greedy. Intervals are Wilson at 95 %, paired comparisons are two-sided exact sign tests over the discordant pairs; the Wilson interval is Chapter 4's and the exact paired sign test is Chapter 15's, both reused unchanged. The human labels are mine, applied to sixty answers under the written rule quoted in the text. Read every magnitude here as a property of a half-billion-parameter model and every method as transferable: a bigger model moves all the numbers up and moves none of the instruments.