Build an Agent Harness: The Loop and Its Five Ways Out

Chapter 14 could not be written against a real provider, because you cannot ask one for a 429 at a chosen moment. This chapter has the same problem in a different shape: you cannot ask a real model to run away, or to request the identical tool twice in a row, on demand and reproducibly.

So the first program is a scripted provider: an endpoint with the shape of a chat completions API whose reply is a function of the turn index and of what the tools have returned so far. It counts tokens with a real byte-pair encoder, so the money below is arithmetic rather than decoration.

const SCRIPTS = {
  // A well-behaved task: list, read, answer.
  plan: (t) =>
    t === 0 ? asks(call("c1", "list_files", {}))
    : t === 1 ? asks(call("c2", "read_file", { path: "errors.log" }))
    : text("errors.log mentions a timeout: worker 7 timed out after 30000 ms."),

  // Never declares itself done.
  runaway: (t) => asks(call(`c${t}`, "list_files", {})),           

  // Guesses a file name, then corrects itself IF it was told what happened.
  recover: (t, all) =>
    t === 0 ? asks(call("c1", "read_file", { path: "timeout.log" }))
    : /Call list_files/.test(all)                                  
      ? (t === 1 ? asks(call("c2", "list_files", {}))
        : t === 2 ? asks(call("c3", "read_file", { path: "errors.log" }))
        : text("errors.log mentions a timeout."))
      : text("I could not read the file, so I do not know."),
};

const turn = messages.filter((m) => m.role === "assistant").length;             
const toolText = messages.filter((m) => m.role === "tool").map((m) => m.content).join("\n");
const message = SCRIPTS[scenario](turn, toolText);

Two lines carry the design. The turn index is derived from the conversation, not held in a variable, so the provider is stateless and a run can be killed and resumed against it. And recover reads the tool results before deciding: a scripted model that reads its own transcript is the minimum needed to measure whether the harness gave it anything worth reading.

The catalogue is Chapter 18's, four tools over three files: list_files, read_file, delete_file — marked needsApproval — and scan_archive, which is slow on purpose.

Here is the whole idea, before any of the parts that make it survivable.

while (true) {
  const reply = await callModel(base, messages, tools, signal);
  messages.push(reply.message);

  const calls = reply.message.tool_calls ?? [];
  if (!calls.length) return reply.message.content;          

  for (const c of calls) {
    const tool = byName.get(c.function.name);
    const result = await tool.run(JSON.parse(c.function.arguments));
    messages.push({ role: "tool", tool_call_id: c.id, name: c.function.name, content: result });
  }
}

Point it at the scripted provider and it does exactly what it looks like it does:

plan, cap 20    turns=3  tools=2  in=815  out=70  cost=$0.002470  ms=89  status=completed
   answer: "errors.log mentions a timeout: worker 7 timed out after 30000 ms."
   per-turn prompt tokens: 204, 269, 342

Three turns, two tool executions, a quarter of a US cent. Note the last line: 204, 269, 342. Every turn resends everything before it, which is Chapter 16's quadratic bill arriving in a place where nobody typed anything. The rest of this chapter is what happens when that line does not stop growing.

Point the same loop at the runaway script — a model that asks for a tool every single turn and never emits prose — and the marked return never fires. There is no other exit. The program runs until the process dies or the credit card does.

The fix is one line, it is the first control the literature recommends,⁵ and everybody writes it eventually. What almost nobody does is measure what it is worth:

Read the last two rows together. Doubling the cap from 50 to 100 did not double the cost; it multiplied it by 3.7. Input tokens went from 88,649 to 337,299, a factor of 3.8, because turn $n$ carries every previous turn with it and the total is $\Theta(n^2)$. A turn cap is not a linear dial. It is a dial on the square root of your worst case, which is why raising it from 20 to 100 "just to be safe" is a decision worth pricing before you make it.

The trouble with a turn cap is that a turn has no fixed price. Twenty turns over a short transcript cost $0.038 above. Twenty turns with a 200-tool catalogue, a retrieved document set and forty messages of history cost hundreds of times that, and the cap does not know. What the operator wants to bound is the bill.

So the loop counts money, using Chapter 16's computeCost against the rates read there — $2.00 per million input tokens and $12.00 per million output, for the model priced throughout this course:

const PRICE_IN = 2.0 / 1e6, PRICE_OUT = 12.0 / 1e6;
export const cost = (u: Usage) => u.prompt_tokens * PRICE_IN + u.completion_tokens * PRICE_OUT;

// at the top of every iteration, before asking the model anything:
if (state.turns >= opts.limits.maxTurns) return stop("max_turns_exceeded", { type: "max_turns" });
if (state.costUsd >= opts.limits.maxBudgetUsd) return stop("budget_exceeded", { type: "max_budget" }); 

// ...and once the reply is back, before anything else happens with it:
state.costUsd += cost(reply.usage);

Same runaway script, no turn cap at all, three budgets:

Two things are worth naming. First, the budget buys a different number of turns each time, which is the point: it is bounding the thing the operator cares about, and letting the turn count fall where the transcript puts it. Second, every row overshoots. The budget was $0.010 and $0.010780 was spent, because the check runs before a turn and the price of a turn is not known until it is over. You cannot bound spend exactly; you can bound it to within one turn's cost. Say so in the interface rather than pretending, and put the check before the call so the overshoot is one turn and not two.

By now the loop has three exits, and the shape of the remaining chapter is visible. A production run ends in exactly one of five ways, and they are not variations of each other:

Collapsing these into one boolean is the most common design mistake in this file, and it is expensive in a specific way: three of the five are resumable and two are not. An agent that hit its turn cap has a valid transcript, a real partial result and a next step; an agent that got a 401 has none of those. So the harness records the reason as data:

export type RunStatus =
  | "running" | "completed" | "failed"
  | "max_turns_exceeded" | "budget_exceeded" | "interrupted";

export type Interruption =
  | { type: "approval"; callId: string; toolName: string; args: unknown }
  | { type: "max_turns" } | { type: "max_budget" }
  | { type: "cancelled"; reason: string };

Chapter 18 ended on a claim without a number: hand a tool's error back to the model as a tool result rather than raising it, and the model usually fixes itself. Here is the number.

One failure, three policies. The scripted model guesses a file that does not exist; the tool throws no such file: timeout.log. Call list_files to see what exists.

The third row costs 4.7 times the first and is the only one that answers the question. And the second row is the interesting one, because it is what most codebases actually do: the error was caught, the loop survived, the model was told that something failed and not what, and it gave up politely. The difference between rows two and three is not error handling. It is a sentence written for a reader.

The harness therefore treats a thrown tool as data, and makes the wording a policy:

} catch (err: any) {
  if (signal.aborted) return stop("interrupted", { type: "cancelled", reason: String(signal.reason) });
  if (opts.toolErrorsAreFatal) { state.error = err.message; return stop("failed"); }
  result = (opts.toolErrorText ?? ((e: Error) => `Error: ${e.message}`))(err);   
}

Chapter 18 also warned about the other side, and it too has a price. Point the loop at a tool that fails for a reason no message can fix — a read the process is not allowed to perform — and the model retries it forever:

read a file the process may not open   turns=12  toolruns=11  in=7,079  cost=$0.018622
                                      status=max_turns_exceeded   answer=""

Eleven identical executions of a call that cannot succeed, 5.2 times the cost of the run that recovered from a fixable one, and nothing at the end. Errors are context; a permanent error is context that poisons the rest of the run. The distinction is Chapter 14's status triage moved one layer up: an error the model can act on goes back into the transcript, and an error it cannot should stop the run with a reason. The turn cap is what stands between you and the second case today, which is a floor and not a fix.

Now the failure most people assume cannot happen. Models repeat themselves. Ask any loop to run long enough and you will see the identical tool with the identical arguments on two consecutive turns.

Measured against the baseline of the same task without the repeat:

The duplicated call cost $0.001050 extra, a 75 % increase, and here is the part that surprises people: caching the result recovered none of it. Deduplication saved the tool execution and not the turn, because by the time your code notices the repeat the model has already been paid for asking. The saving is real when the tool is slow, rate-limited, or billed by the call — and it is zero on the line item that grew.

There is a worse version. Apply the same cache to a tool that writes, and the second call silently does not happen:

naive cache on every tool        3 turns, 1 tool run,  files deleted: ["access.log"]
cache only on read-only tools    3 turns, 2 tool runs, files deleted: ["access.log","access.log"]

Which of those is correct? Neither, knowably. The protocol says these are two calls: they carry two different tool_call_id values. The arguments say they might be one. A harness that decides by comparing argument strings will one day swallow the second of two identical, intended charges — and Chapter 14 already named the only mechanism that resolves this honestly, which is an idempotency key generated per logical operation by the layer that knows what the operation is. Until the tool carries one, the defensible default is the read-only gate above: cache reads, execute writes, and let the write's own idempotency handle the rest.

if (opts.dedupe && (tool.readOnly || opts.dedupeAll) && seen.has(signature)) {   
  state.messages.push({ role: "tool", tool_call_id: c.id, name: c.function.name, content: seen.get(signature)! });
  continue;
}

The destructive script lists the files and then asks to delete one the task never mentioned. Nothing in the loop so far would stop it.

A tool marked needsApproval does not fail and does not proceed. It stops the run and returns control, with everything a person needs to decide:

if (tool.needsApproval && !state.approved.includes(c.id)) {
  trace(state.runId, "approval_required", { toolName: tool.name, args: c.function.arguments, callId: c.id });
  return stop("interrupted", { type: "approval", callId: c.id, toolName: tool.name, args: JSON.parse(c.function.arguments) });
}

stopped at turn 2: interrupted / approval -> delete_file({"path":"access.log"})
files deleted so far: []
approve -> total turns=3  deleted=["access.log"]  "Deleted access.log to free space."
reject  -> total turns=3  deleted=[]              "I did not delete anything: you declined the deletion."

That is the whole mechanism, and the reason it is a return rather than a callback is the next section: between the stop and the verdict, the process may not exist any more.

But first, the measurement nobody expects. A rejection is not the absence of a result — the transcript has a slot keyed by tool_call_id and something has to go in it. Run the same rejection twice, changing only what that something says:

rejected with a reason   deleted=[]  the agent then told the user:
                                     "I did not delete anything: you declined the deletion."
rejected with nothing    deleted=[]  the agent then told the user:
                                     "Deleted access.log to free space."

Nothing was deleted in either run, and in the second the user is told it was. The permission system worked perfectly; the report is a lie. It is the same mechanism as the tool-error table, arriving somewhere that matters much more — a human said no, the action was correctly blocked, and the agent's summary contradicts reality because the refusal was never written down where the model reads. The rule that falls out of this is short: whatever your code decides about a tool call, write the decision into the transcript in words. Chapter 30 comes back to this from the security side, where it is the difference between an audit trail and fiction.

An approval takes minutes or hours. A deploy takes seconds. If the run lives in a local variable inside an HTTP request, every restart is a lost run and every approval is a race.

So the run is not a closure. It is a plain serialisable object — messages, turn count, cost, status, interruption, the list of approved call ids — and the loop is a pure function over it. That single constraint is what makes persistence a one-line concern:

export const save = (s: RunState, dir: string) => writeFileSync(`${dir}/${s.runId}.json`, JSON.stringify(s));
export const load = (dir: string, runId: string) => JSON.parse(readFileSync(`${dir}/${runId}.json`, "utf8"));

The correctness question is not saving. It is what happens on the way back in, and the naive answer double-charges you. If the process died after the model asked for a tool but before the result was written, a resume that starts by calling the model again pays for a turn it already has — and if it starts by re-running the tools, it performs a write twice.

The fix is to make the loop begin by asking the transcript what is outstanding:

export function pending(state: RunState): ToolCall[] {
  const answered = new Set(state.messages.filter((m) => m.role === "tool").map((m) => m.tool_call_id));
  const last = state.messages.at(-1);
  if (last?.role !== "assistant") return [];
  return (last.tool_calls ?? []).filter((c) => !answered.has(c.id));    
}

Every iteration drains pending first and only asks the model when there is nothing outstanding. Resume becomes the same code path as the normal one, and so does approval — an approved call is simply a pending call that is now allowed to run. Kill the process mid-task and restart it:

process died after turn 2. tool runs so far: list_files, read_file:errors.log
restored from disk: turns=2  cost=$0.001570  messages=6  status=running
resumed and finished: turns=3  cost=$0.002470  status=completed
tool runs across BOTH processes: list_files, read_file:errors.log

Two tool executions across two processes for a task that needs two, and the final cost is identical to the run that never crashed. Cost accumulates across the restart because it was in the state, not in a variable.

scan_archive takes three seconds here and stands in for the tool that takes three minutes in production. Two things are missing while it runs: the user has no idea anything is happening, and the Stop button does nothing.

Both are the same fix, and it is Chapter 14's AbortSignal pushed one level deeper. The signal is not only for the fetch — it is passed into the tool, and a well-written tool honours it:

result = await tool.run(JSON.parse(c.function.arguments), {
  signal,                                                                     
  progress: (label) => { trace(state.runId, "tool_progress", { toolName: tool.name, label }); opts.onProgress?.(label); },
});

progress: scanned 200 of 1200 files  (t+506 ms)
progress: scanned 400 of 1200 files  (t+1007 ms)
no cancellation:            stopped after 3,015 ms, status=completed
user presses Stop at 1.2 s: stopped after 1,202 ms, status=interrupted, reason="user pressed Stop"

Two milliseconds from the click to the stop, because the sleep inside the tool listens to the same signal that the fetch does. Thread it only into fetch and the identical Stop button waits three seconds — the length of the tool — and the run "cancels" after the work it was cancelling has already finished. Cancellation that is not plumbed all the way down is a spinner that says the right word.

The harness emits one line per event, and the vocabulary is small enough to memorise: turn, tool_start, tool_progress, tool_result, approval_required, run_stopped.

{"runId":"n1","type":"turn","turn":1,"prompt_tokens":204,"completion_tokens":23,"total_tokens":227,"costUsd":0.000684,"finish":"tool_calls"}
{"runId":"n1","type":"tool_start","toolName":"list_files","args":"{}","callId":"c1"}
{"runId":"n1","type":"tool_result","toolName":"list_files","ms":1,"ok":true}
{"runId":"n1","type":"turn","turn":2,"prompt_tokens":269,"completion_tokens":29,"total_tokens":298,"costUsd":0.00157,"finish":"tool_calls"}
{"runId":"n1","type":"approval_required","toolName":"delete_file","args":"{\"path\":\"access.log\"}","callId":"c2"}
{"runId":"n1","type":"run_stopped","status":"interrupted","reason":"approval","turns":2,"costUsd":0.00157}

Three properties make this a trace rather than logging. Every line carries the run id, so a run that spans three processes and two days is one query. Every turn line carries its own token counts and the running cost, so "why did this run cost forty dollars" is answerable after the fact instead of reproducible only in theory. And run_stopped carries the reason, which is the field that turns a support ticket into a one-line answer: an agent that stopped at the budget and an agent that crashed look identical from the outside and need opposite responses.

Chapter 13 measured time to first token on hardware you own. Chapter 14 measured it through a socket. An agent multiplies it, and the multiplier is a number nobody chose:

The same three-turn task, changing only the provider's latency:

The harness itself contributes fifteen milliseconds to a three-turn run. Everything else is $N$ multiplied by a number you do not control — set inside a serving scheduler that is batching your request with strangers' requests⁶ — and $N$ is chosen by the model. This is why the streaming of Chapter 14 matters more here than in a chat and helps less: you can stream the final turn, and the four turns before it are silence unless the harness emits progress. It is also the whole argument for the tool_progress event above — in an agent, the honest unit of feedback is not the token, it is the step.

Everything above ran against a scripted provider, which proves the harness and proves nothing about models. So change one line — the seam from Chapter 14, LLM_BASE_URL — and point the identical code at a local Qwen2.5-0.5B-Instruct with the same four tools. Six tasks over the same three files:

turns=2 tools=1 wall= 15,260ms  Which file mentions a timeout?      -> "The file timeout.txt does not exist..."
turns=2 tools=1 wall= 13,037ms  How many files are in the directory? -> "There are three files..."
turns=2 tools=1 wall= 10,121ms  Read notes.txt and tell me what it says. -> "Remember to rotate your logs."
turns=2 tools=2 wall= 21,290ms  List the files and then read each one.
turns=2 tools=1 wall= 10,698ms  Which file is the largest?          -> "The largest file is access.log."
turns=2 tools=1 wall= 12,490ms  Is there a file about rotating logs?
TOTAL turns=12  toolruns=7  wall=82,896ms  mean turn=6,908ms

Three findings, and the third is the reason this section exists.

Every single task finished in exactly two turns. The turn cap never fired, the budget never fired, and the loop's only exit was the model producing prose. A half-billion-parameter model does not iterate; it answers on its second breath whether or not it has what it needs. The turn count is a property of the model, not of your loop.

The mean turn took 6,908 milliseconds, so the latency table above is not a toy: at this size a hypothetical eight-turn run is nearly a minute of wall clock with nothing on screen.

And the answers are wrong. The largest file is errors.log; the model listed the files, never read them, and named one anyway. The first task guessed a file name, was told it did not exist, and concluded. The harness executed flawlessly in all six runs. A harness makes an agent governable, not correct — Chapter 29 is how you find out which, and Chapter 30 is what it costs when nobody did.

One tool in the catalogue can have another run behind it. The interface is Chapter 18's — a schema and an endpoint — and an entire agent fits behind it because that interface is narrow:

const research: Tool = {
  name: "research",
  description: "Investigate one question and return a short summary.",
  parameters: { type: "object", properties: { question: { type: "string" } }, required: ["question"] },
  readOnly: true,
  async run(args, ctx) {
    const child = newRun(RESEARCH_SYSTEM, args.question);        // its own transcript
    const out = await run(child, researchTools, { base, limits: { maxTurns: 6, maxBudgetUsd: 0.05 }, signal: ctx.signal });
    return out.output ?? "no result";
  },
};

Three things are already right in those ten lines and all three are consequences of decisions made above: the child has its own window, so the parent's transcript receives a summary rather than everything the child read; it has its own limits, so a runaway child cannot spend the parent's budget; and it inherits the signal, so one Stop cancels the tree. Why a clean window is the point rather than a side effect is Chapter 24; the five orchestration patterns — prompt chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser — and the handoff are Chapter 25.

Nothing above should be read as an argument against libraries. Measured on 7 September 2026, for the month ending 29 August:⁷

The reason this course writes the loop by hand instead of teaching one of them is declared rather than implied, and it is measurable. In the twelve months to 7 September 2026, ai published 945 versions and moved from major 5 to major 7, and its agent class is still exported as Experimental_Agent; langchain published 132 versions in the same window; @openai/agents published 83 and is still on 0.x, fifteen months after its first release.⁷ A chapter written against any of those APIs is stale within a season, and this one is published in thirty-three languages, so every re-edition costs the whole translation. What is underneath all of them does not move: a loop, a stopping rule, a catalogue, an executor, some state.

And the reference implementation agrees with this chapter about the part that matters. In ai version 7.0.93 the loop's exit is not a number — it is stopWhen, a list of predicates, of which a step count is merely one:³

type StopCondition<TOOLS extends ToolSet> = (options: { steps: Array<StepResult<TOOLS>> }) => PromiseLike<boolean> | boolean;
declare function isStepCount(stepCount: number): StopCondition<any, any>;   // exported as stepCountIs

Stopping is plural in the most-used implementation of this loop, for the same reason it is plural in the hundred and ninety-six lines above.

You now have a harness: a loop, a catalogue, an executor, five ways out, a persisted run, a signal that reaches the tools, and a trace with a run id on every line. Chapters 24, 25, 29 and 30 build on this file, and 26 to 28 on what it can reach.

It has one problem left, and the measurements above have been pointing at it the whole way. Look at the runaway table once more: 3,431 input tokens at eight turns, 337,299 at a hundred. Look at the working run: 204, 269, 342. Every turn resends the whole transcript, so an agent's context fills up with its own history — and the model is worse at using the far end of a long window than the near end, which is why a good agent at turn five is a confused one at turn forty.

A turn cap does not fix that. It just stops you from paying to watch it happen. What fixes it is deciding, on every single turn, which tokens deserve the window: what to compact, what to move out to a note the agent can fetch, what to hand to a subagent with a clean window, and which tool definitions are worth their permanent tax. Chapter 24 measures where the window actually goes — and the surprise is that it is not the conversation.

Every number in this chapter came out of the two servers described above, on Node 22 over a loopback interface: a scripted provider counting tokens with the o200k_base encoding, and Qwen/Qwen2.5-0.5B-Instruct behind an endpoint of the same shape, greedy decoding, on CPU. Costs are computed from measured token counts at the rates Chapter 16 read on 6 September 2026 — $2.00 per million input tokens and $12.00 per million output — and no request in this chapter went to a paid endpoint. The local model's answers are a small model's answers; read them as evidence about the loop, which is identical either way, and not as a benchmark of what current models do.