Context Engineering: Why Your Agent Gets Dumber at Turn 40

Anthropic drew the line in September 2025 and the two sentences belong side by side. Prompt engineering is "methods for writing and organizing LLM instructions for optimal outcomes". Context engineering is "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts".¹

The operative difference is when, and by whom. A prompt is authored once, by a person, and reviewed. A context is assembled on every call, by code nobody is looking at, out of material nobody wrote by hand: forty turns of history, six tool results, four retrieved passages, a user profile, twelve JSON schemas. Chapter 15 measured what better instructions buy. This chapter is about the other ninety per cent of the tokens, which arrive by themselves.

The same document names the resource that all of them spend: models "have an 'attention budget' that they draw on when parsing large volumes of context. Every new token introduced depletes this budget by some amount". And it names the symptom: "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases" — context rot.¹

That last sentence is a claim about behaviour, which means it can be checked, and the table at the top of this page is the check.

Forty lines against the local endpoint from Chapter 22 — a small Python server holding Qwen2.5-0.5B-Instruct on the CPU and speaking the chat-completions shape, so the loop stays TypeScript and the tensors stay on the far side of the port.

const DEPTHS = [0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1];

for (const d of DEPTHS) {
  const slot = Math.round(d * (N - 1));
  let hits = 0, other = 0;
  for (let t = 0; t < TRIALS; t++) {
    const recs = buildRecords(N, 1000 + t);          // 25 unique tickets
    const gold = recs[Math.floor(rng(7 + t)() * N)]; // a different one each trial
    const rest = recs.filter((x) => x.ticket !== gold.ticket).slice(0, N - 1);
    const lines = [...rest.slice(0, slot).map((x) => x.line),   
                   gold.line,                                   
                   ...rest.slice(slot).map((x) => x.line)];     
    const r = await complete(prompt(lines, ask(gold.owner)), { maxTokens: 12 });
    const said = /\d{4}/.exec(r.text)?.[0];
    if (said === String(gold.ext)) hits++;
    else if (said && recs.some((x) => String(x.ext) === said)) other++;
  }
}

The other counter is what turns a disappointing result into a useful one: when the model is wrong, is it lost or is it confident?

The answer is confident. Across the eight non-first positions, 136 of the 205 wrong answers were another ticket's extension — a real four-digit number, correctly formatted, read off the wrong line. At slot 1 only one of the five misses was; at slot 7, twenty-one of twenty-six were.

That distinction is what matters in production. A model that says I cannot find it is a bug you notice; a model that returns a neighbouring row's number is a bug you ship, because on screen the two look identical. It is the failure Chapter 19 built verifiable citations against, arriving from inside the prompt instead of from the index.

Position is one axis. Length is the other, and easier to test: keep the answer in the middle and grow the list.

One record and 97 tokens: 90 %. Three records and 159 tokens: 55 %. Eight records and 315 tokens: 15 %, and from there flat and low all the way out to 140 records and 4,477 tokens. The entire collapse happens between the first and the eighth line of a list.

The last column is everything that is neither the right extension nor another record's, which with a single record on the page is the only place a wrong answer can land. The two misses at one record are worth reporting rather than rounding away, because neither was a refusal: one answered 5806 to a register whose only line says 5805. At 97 tokens with a single candidate this model still miscopies a digit twice in twenty, and that is the floor everything else is measured against.

Two things follow. A bigger context buys the right to send more, not the certainty of being read: this model has a 32,768-token window and a working range, on this task, of a few hundred tokens. And there is no threshold, no cliff, no "context full" state — degradation is under way at the third record and complete by the eighth, at one per cent of the window. Whatever a context limit is, it is not what governs this.

Two mechanisms are usually offered. The first is the arithmetic from Chapter 9, which Anthropic states in the same terms this course does: models "are based on the transformer architecture, which enables every token to attend to every other token across the entire context. This results in n² pairwise relationships for n tokens".¹ Attention over a longer sequence is not the same operation applied to more material; it is one fixed budget of probability mass spread over more competitors. The second is training: models see far more short sequences than long ones, so long-range positional patterns are the least practised part of the network. That is an argument, not a measurement, and this chapter cannot settle it.

What is settled is the shape, and has been since 2023. Liu et al. tested multi-document question answering and key-value retrieval across model families and sizes and found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".² Chapter 15 took its position rule from that paper; Chapter 19 took from it the reason twenty retrieved chunks can score worse than four. The practical form of the fact is the only sentence here you should act on: this takes five minutes to measure on your own model with your own data, and no published curve substitutes for yours.

Ask a team what fills their agent's context and you get an estimate, because no API returns the answer: the response gives you prompt_tokens, one number for all of it.

You can recover the breakdown with four counts and three subtractions — the whole rendered prompt, the same without tool definitions, the system message alone with and without them, and everything with the tool results removed:

async function buckets(messages: Msg[]) {
  const sys = messages.slice(0, 1);
  const withoutResults = messages.filter((m) => m.role !== "tool");
  const [total, sysWithTools, sysNoTools, noResults] = await Promise.all([
    countPrompt(messages, CATALOGUE),        // everything
    countPrompt(sys, CATALOGUE),             // system + scaffolding + schemas
    countPrompt(sys),                        // system + scaffolding
    countPrompt(withoutResults, CATALOGUE),  // everything but tool output
  ]);
  return {
    system: sysNoTools,
    tools: sysWithTools - sysNoTools,                                  
    toolResults: total - noResults,                                    
    conversation: total - sysWithTools - (total - noResults),          
    total,
  };
}

countPrompt applies the model's own chat template before tokenizing, which matters more than it sounds: your text is not what gets counted. Role markers, the tool-calling preamble and the schema rendering are all tokens you pay for and never typed. Chapter 7 built a tokenizer and Chapter 16 counted with js-tiktoken; here the count comes from the same model that will read the prompt, which is the only count that is exactly right.

Now run a real agent through it: forty turns of an incident investigation, twelve tools, a fake operations environment returning realistic log dumps and metric series.

Read the first row against the last.

At turn 1 the prompt is 2,547 tokens and 71 % of it is tool definitions. The system prompt is 3 %. What the user typed is 6 %. The agent has done nothing yet and is already carrying 1,817 tokens of JSON schema.

By turn 40 the prompt is 10,632 tokens and the shares have inverted: definitions 17 %, conversation 29 %, tool results 53 %. Tool output overtook the definitions at turn 5; the conversation did not overtake them until turn 25, so for the first sixty per cent of the session the tool catalogue was bigger than everything that had been said.

Then the total. Across 57 model calls the run billed 370,291 input tokens for a final context of 10,632 — the last prompt paid for about thirty-five times over, which is Chapter 16's quadratic with an agent's multiplier on top. Of those 370,291, 103,569, or 28 % of everything billed, were the twelve tool definitions, resent byte-identical on every call.

The tool catalogue is the largest fixed cost in an agent and it is invisible, because you never see it: you pass an array of objects and the provider renders it into the prompt for you. Measured, on the same twelve tools:

system prompt + chat scaffolding, no tools:        85 tokens
all twelve definitions:                          1,817 tokens
  of which fixed tool-calling scaffolding:         126 tokens
three tools instead of twelve:                     605 tokens
same twelve, one-sentence descriptions,
  no parameter prose:                            1,291 tokens  (-29 %)

Per tool the marginal cost runs from 80 tokens for get_current_time, which takes one string, to 263 for search_tickets, which takes four parameters with an enum and a sentence of guidance each. That is the exchange rate behind Chapter 18's central advice that the description is the API: a good description costs about a hundred tokens on every request for the rest of the agent's life. Three consequences.

A tool you do not use still bills. The agent called seven of the twelve. The other five cost 697 tokens on each of the 57 requests — 39,729 in all, more than a tenth of everything the run was billed, for capabilities it never touched. One of the five carries the sharpest detail in the trace: the model tried three times to call read_log, which does not exist. The tool it wanted was search_logs, the second most expensive definition in the catalogue at 237 tokens. It paid for that definition 57 times, never used it, and never found its name.

Trimming prose is the cheapest optimisation available, and it is a trade. Cutting descriptions to one sentence and dropping parameter documentation saved 526 tokens per call, 29 per cent, without touching a line of logic — and made the model call the tools worse, which is what Chapter 18 measured. The point is that both sides of that trade are now in the same unit.

At some scale, sending definitions at all stops making sense. Anthropic put a number on it in November 2025: a large set of connected servers means processing "hundreds of thousands of tokens" of definitions before the request is read, and replacing that with code execution — the agent discovering and loading only the definitions it needs — "reduces the token usage from 150,000 tokens to 2,000 tokens, a time and cost saving of 98.7%".³ Same idea as the rest of this chapter, applied to schemas instead of history: keep the index, resolve the entry on demand.

Two things were planted in that forty-turn transcript. At turn 2, before any real work, the user states a standing rule: any ticket you open must be filed under my employee number, 4417. At turn 19, in the middle of the incident, a fact: the affected shard is pay-shard-7, confirmed by the payments team. At turn 40 the user asks the agent to open the incident ticket, which needs both. Each probe is asked in six different phrasings and scored out of six — greedy decoding is deterministic, so one call gives an unrepeatable yes or no and six give a rate.

The transcript is then replayed under seven context policies. Replayed rather than re-run, deliberately: the messages, the tool calls and the tool results are byte-identical in all seven, so the only variable is what each policy chose to keep. Chapter 16 showed why a sliding window is a bad economic move, because it destroys the cacheable prefix. Here is what it does to behaviour:

The compaction rows include what compacting cost: 18,581 input tokens for seven summaries and 3,392 more for the note-taker. The control row is there so a zero can be read as a zero — with the two messages alone in a 1,981-token prompt this model answers both probes perfectly, so no row is the task being too hard.

Full history remembers, and is the most expensive thing on the table: 370,291 input tokens for a session whose durable content is two sentences.

That answers a question the opening left open. Why does a 10,632-token transcript hold a fact that an 853-token register loses? Because length is the wrong variable. The register holds twenty-five four-digit extensions in twenty-five identical sentences — twenty-four near-perfect decoys for the one you want. The transcript holds exactly one employee number and one shard name. Context rot is interference before it is volume, which is why 136 of the 205 wrong answers up there were a neighbour's value. The useful question about a window is not how long it is; it is how many things in it look like the answer.

The sliding window is 57 % cheaper and has lost the incident. The employee number survives only because the agent had repeated it into the recent turns. The shard, stated once at turn 19, is not in the last twelve messages — and the model does not say so. Asked six times it answered "the affected payment shard is shard 4417", reaching for the employee number, the only other identifier left in its window, and twice "pool", lifted out of the string pool_exhausted in a log line.

Compaction is cheap and lost the same fact. Seven summaries, written by the model under an explicit instruction to keep identifiers, numbers, standing instructions and open questions, and pay-shard-7 is in none of the ones that mattered; the six guesses were shard 1, pay_shard_1 and pool. Compaction does not fail loudly. It produces a fluent, plausible, much shorter session that has quietly dropped one line.

Three rows scored 0/6 on the turn-19 fact — the sliding window, compaction, and compaction with notes. Eighteen wrong answers between them, and not one of them was "I do not know."

Then the row that ought to be embarrassing. Keeping the user's own forty messages verbatim, plus the last four turns in full and nothing else, costs 168,550 tokens — 54 % less than full history — and answers both probes as well as full history or better. No summariser, no note-taker, no second model: a filter on role === "user". The user's words are the cheapest high-value tokens in an agent's window, and most designs discard them with everything else.

The last two rows are the opening table again, inside the agent. The same pinned block, moved from the system message to the end of the prompt: 5/6 becomes 6/6. On six trials that is not a significant difference and is not offered as one — it is offered as a reminder that where is a parameter you are setting whether you know it or not.

The four strategies below are Anthropic's, in its order, though only the last three are its long-horizon list.¹ All four are variations on one instruction: do not carry what you can fetch, and do not carry raw what you can carry compressed.

const elide: Policy = (h) => [SYSTEM, ...h.flatMap((turn, ti) =>
  turn.map((m) => (ti < h.length - 4 && m.role === "tool"
    ? { role: "tool", name: m.name,
        content: `[${m.name} result from turn ${ti + 1}, ${m.content.length} chars, ` +
                 `elided; call ${m.name} again with the same arguments to re-read it]` }
    : m)))];

const r = await complete([
  { role: "system", content:
      "You keep a durable note file for a support session. Given one user message, " +
      "output one short note ONLY if it states a standing rule, an identifier or a fact " +
      "that must survive the rest of the session. Otherwise output exactly NONE." },
  { role: "user", content: `Turn ${i + 1}: ${user}` },
], { maxTokens: 40 });
if (!/^none\b/i.test(r.text.trim())) notes.push(`turn ${i + 1}: ${r.text.trim()}`);

The sub-agent is not in the table above because it does not run for forty turns: it runs once, in a window somebody scoped for it. Given the system prompt, turns 17 to 19 and nothing else — 2,737 tokens — it answered the shard probe 6/6, better than every policy in the table, and the employee probe 0/6, because that number is not in the three turns it was handed.

That is sub-agents in two numbers: a clean window is not intelligence, it is scope, and the scoping is done in advance by code that already has to know which turns matter. One more thing in those answers is worth keeping. This was the only policy that replied "None available" instead of inventing something. A model with a small, coherent context knows what it is missing; a model with a large, noisy one does not.

Almost every confused conversation about agent memory is three mechanisms wearing one word. They have different lifetimes, owners and failure modes, and a system that keeps them in the same place has a problem it has not noticed yet.

The academic framing is CoALA's, which organises language agents around "modular memory components" and separates working memory from episodic, semantic and procedural stores.⁴ MemGPT takes the same idea literally, borrowing virtual memory from operating systems: a fast tier inside the window, a slow tier outside it, and the model itself moving data between them with function calls.⁵ Both force the question a product has to answer anyway — not how much can I keep, but which store does this belong in, and when does it expire.

The practical test is one question per fact: what should still be true tomorrow? A tool result from turn 12, nothing. A summary of the session, until the session ends. That the user's employee number is 4417, until they change jobs. Three answers, three stores.

You can now measure what is in a window, decide what stays in it, and tell the difference between an agent that forgot something and one that was carrying it and did not look.

The last of the four strategies is the one that does not fit here. A sub-agent is not a context policy, it is a second agent, and the moment there are two you have to decide what passes between them and which is in charge. Chapter 25 is that: the five orchestration patterns and where each of their names actually comes from, the two topologies that get mixed up — asking a sub-agent and getting an answer back, against handing it the conversation and not getting it back — and the measured finding that on the task it prices, the simpler arrangement wins — followed by the test for when it stops winning.

It also inherits exactly what this chapter just measured. A sub-agent returns a summary. A summary is a compaction you did not write, produced by a model whose window you cannot see, and the parent has no way to tell a good one from a confident wrong one — the same distinction that separated 84 % from 19 % at the top of this page, and that turned eighteen missing facts into eighteen invented ones. So: when the sub-agent is wrong, what exactly does the parent get to look at?

Every number here was produced on this machine and none was estimated. The model is Qwen2.5-0.5B-Instruct in float32 on the CPU with greedy decoding, served over loopback by a small Python endpoint that speaks the chat-completions shape and exposes a token-count route — Chapter 14's seam again, tensors on the Python side and the loop on the TypeScript one — so every count is that model's own tokenizer applied to its own chat template. The position table is 288 calls, nine positions by thirty-two trials with a different ticket each trial; the length table is 140 calls; the agent run is 57 model calls over 43 minutes of wall clock; the policy table is that one transcript replayed under seven policies. Intervals are Wilson's, from Chapter 4. No paid API was called, which is also why there is not one price in the chapter: the token counts are exact and the rates you would multiply them by are Chapter 16's.