The Context Window, Tokens and the Bill, Measured

The single most expensive misconception in this business is that a model remembers a conversation.

It does not, and the mechanism from Chapter 13 says exactly why. A transformer's state during generation is the KV cache: the keys and values computed for every token in the sequence. That cache lives for the duration of one request. When the request ends, the process that held it is free to serve somebody else, and the cache is gone. There is no per-user store on the other side, and no session.

So the next request has to arrive carrying everything the model is supposed to know, and the model rebuilds that state by running a forward pass over the entire prompt before emitting a single new token. Chapter 15 called the prompt "the entire state". This is the physical reason: the prompt is the complete state because nothing else survives the call.

The context window is the maximum length of that prompt plus its answer. It is a ceiling on how much state you can rebuild, not a container that holds anything between requests. Calling it "the model's memory" gets the direction of causality backwards — you are not filling a memory, you are paying to re-establish one.

That is where the twenty-two comes from. Turn $n$ carries all $n-1$ previous turns, so the total input across a conversation of $n$ turns is the sum of a growing series, which is quadratic:

where $s$ is the system prompt and $h_i$ the history at turn $i$. Fitting the measured cumulative input to $an^2 + bn$ over the forty turns gives $60.22\,n^2 + 432.25\,n$, which predicts 113,645 tokens at turn 40 against 112,617 measured. The quadratic term dominates and the linear term is what the user actually typed.

The consequence is the sentence to take away from this chapter: your bill grows with the square of the conversation, not with the last question. The same forty questions asked with no history at all cost $0.066036. Keeping the history cost $0.274386. History multiplied the bill by 4.2, and it will keep multiplying, because the multiplier is the conversation length.

The window is finite for two reasons that pull in the same direction. The first is Chapter 9's: attention compares every token with every other token, so that layer's work grows with the square of the sequence length. The second is memory: the KV cache grows linearly with sequence length, and Chapter 13 did that arithmetic — at long sequences it is larger than the weights.

Both limits have been attacked and neither has been removed. FlashAttention¹ reorganises the computation so it reads and writes far less to high-bandwidth memory, which makes long sequences practical without changing the asymptotic cost. Position Interpolation² and YaRN³ extend a trained model's usable window by rescaling the positional encodings from Chapter 9 rather than retraining. Together they are why windows went from 2K to 1M in five years.

What they did not do is make long contexts free. They made the ceiling higher and the slope gentler. The slope is still there, and it is what the price tiers later in this chapter are measuring.

Nearly every cost calculator on the internet models an API call as input tokens times an input price plus output tokens times an output price. That was true in 2023. It is now wrong in a way that produces bills off by a factor of two or more in both directions.

There are five billable token categories:

Three of those five did not exist as separate lines two years ago, and the two cache lines are the ones people get wrong, because a cache write costs more than ordinary input, not less. You pay a premium to store something so that you can pay a discount to read it back, and whether that trades well depends entirely on how many times you read it.

The reasoning bucket is Chapter 12's, now with a price on it, and it carries a detail worth stating plainly: Google's documentation says pricing "is based on the full thought tokens the model needs to generate, despite only the summary being output from the API."⁴ You are billed for tokens that are never transmitted to you. It is the only bucket whose contents you cannot count, inspect, or verify.

Now the part that makes this a normalisation problem rather than a multiplication problem. Every provider reports these buckets under different names, and — this is the trap — two of them use the same word for two different quantities.

Take one call: 4,837 tokens read from cache, 110 fresh, 142 visible output tokens, 300 reasoning tokens.

// OpenAI-compatible
{ "usage": { "prompt_tokens": 4947,
             "prompt_tokens_details": { "cached_tokens": 4837 },
             "completion_tokens": 442,
             "completion_tokens_details": { "reasoning_tokens": 300 } } }

// Anthropic
{ "usage": { "input_tokens": 110,
             "cache_read_input_tokens": 4837,
             "cache_creation_input_tokens": 0,
             "output_tokens": 442 } }

// Gemini
{ "usageMetadata": { "promptTokenCount": 4947,
                     "cachedContentTokenCount": 4837,
                     "candidatesTokenCount": 142,
                     "thoughtsTokenCount": 300 } }

Look at prompt_tokens: 4947 and input_tokens: 110. Both fields are the input token count for the same prompt. OpenAI's includes the cached tokens; Anthropic's excludes them — its documentation states the identity explicitly, total_input_tokens = cache_read_input_tokens + cache_creation_input_tokens + input_tokens.⁵ Anthropic's input_tokens means "the tokens after your last cache breakpoint".

And look at the output. OpenAI and Anthropic both report 442, which already contains the 300 reasoning tokens. Gemini reports 142 and puts the 300 in a field of its own. Chapter 12 flagged this as an incompatibility between two ways of counting the same work; here is what it costs.

A normaliser is thirty lines and it is not optional:

export interface Usage {
  promptTokens?: number;        // input, NOT cached
  cachedInputTokens?: number;   // read from cache
  cacheWriteTokens?: number;    // written to cache on this call
  completionTokens?: number;    // output
  reasoningTokens?: number;     // billed apart from output (Gemini only)
}

const num = (v: unknown) => (typeof v === "number" && isFinite(v) ? v : 0);

export const fromOpenAI = (raw: any): Usage => {
  const u = raw.usage ?? {}, d = u.prompt_tokens_details ?? {};
  const cached = num(d.cached_tokens), write = num(d.cache_write_tokens);
  return {
    promptTokens: Math.max(0, num(u.prompt_tokens) - cached - write), 
    cachedInputTokens: cached,
    cacheWriteTokens: write,
    completionTokens: num(u.completion_tokens),   // reasoning already inside
    reasoningTokens: 0,
  };
};

export const fromAnthropic = (raw: any): Usage => {
  const u = raw.usage ?? {};
  return {
    promptTokens: num(u.input_tokens),            // already excludes cache
    cachedInputTokens: num(u.cache_read_input_tokens),
    cacheWriteTokens: num(u.cache_creation_input_tokens),
    completionTokens: num(u.output_tokens),
    reasoningTokens: 0,
  };
};

export const fromGemini = (raw: any): Usage => {
  const m = raw.usageMetadata ?? {}, cached = num(m.cachedContentTokenCount);
  return {
    promptTokens: Math.max(0, num(m.promptTokenCount) - cached),
    cachedInputTokens: cached,
    cacheWriteTokens: 0,
    completionTokens: num(m.candidatesTokenCount), // EXCLUDES thinking
    reasoningTokens: num(m.thoughtsTokenCount),    // billed at output rate
  };
};

Run the three payloads above through the three readers and all three produce the same Usage, and therefore the same number: $0.006491. That agreement is the whole point of writing the layer.

Get it wrong and here is what it costs, on the same call:

The third one is the dangerous one, because it fails silently in the direction of good news. Your dashboard shows a reasoning model costing less than half what it costs, and nothing anywhere raises an error.

With the buckets normalised, the cost function is short. The only non-obvious part is the tier lookup, which the next section explains:

export interface Tier { maxPromptTokens: number | null; price: number }
export interface Pricing {
  input: Tier[]; output: Tier[];
  cachedInput?: Tier[]; cacheWrite?: Tier[]; reasoning?: Tier[];
}

const tierPrice = (tiers: Tier[] | undefined, contextSize: number, fallback?: Tier[]) => {
  const table = tiers ?? fallback;
  if (!table?.length) return 0;
  const sorted = [...table].sort(
    (a, b) => (a.maxPromptTokens ?? Infinity) - (b.maxPromptTokens ?? Infinity));
  for (const t of sorted)
    if (t.maxPromptTokens === null || contextSize <= t.maxPromptTokens) return t.price;
  return sorted[sorted.length - 1].price;
};

export function computeCost(pricing: Pricing, usage: Usage): number {
  const fresh = usage.promptTokens ?? 0;
  const read  = usage.cachedInputTokens ?? 0;
  const write = usage.cacheWriteTokens ?? 0;
  const out   = usage.completionTokens ?? 0;
  const think = usage.reasoningTokens ?? 0;
  const contextSize = fresh + read + write;   // the tier depends on the WHOLE prompt
  return fresh * tierPrice(pricing.input, contextSize)
       + read  * tierPrice(pricing.cachedInput, contextSize, pricing.input)
       + write * tierPrice(pricing.cacheWrite,  contextSize, pricing.input)
       + out   * tierPrice(pricing.output, contextSize)
       + think * tierPrice(pricing.reasoning, contextSize, pricing.output);
}

Two design decisions there are worth arguing for. The fallbacks — cache prices falling back to input, reasoning to output — encode what a missing table means: reasoning tokens on Gemini are billed at the output rate, so an absent reasoning price is not zero, it is the output price. And contextSize sums all three input buckets rather than the fresh ones, because the tier is chosen by how long the prompt is, not by how much of it you were charged full price for.

A prompt cache stores the model's computed state for a prefix of your prompt, so a later request with the same prefix skips recomputing it. Four properties follow from the word "prefix" and all four surprise people.

The break-even is small enough to keep in your head, and OpenAI's documentation does the arithmetic: writing a prefix once and reusing it once costs 1.35× its ordinary input cost, against 2× for processing it twice uncached; across ten requests, one write and nine reads cost 2.15× against 10×. One reuse pays for the write. Anthropic lands in the same place: one read for the five-minute cache, two for the one-hour cache.

Now the forty-turn conversation again, with caching on and the prefix stable:

Sixty-eight per cent cheaper, and three numbers in that table repay attention.

The cache does not engage until turn 6. The prompt does not reach 1,024 tokens until then, so the first five turns are billed exactly as before — and the sixth is billed worse, at the 1.25\u00d7 write premium, because it is the turn that fills the cache. The first read arrives at turn 7. The 2,887 uncached tokens in the table are the arithmetic: five turns' worth, not six. Caching is a discount on long prompts, and a short conversation gets nothing from it.

The write premium is $0.002474, which is 2.8 % of the cached bill. Every turn writes its new tail, forty times, and the whole write premium is a rounding error against what the reads saved. The write charge is worth understanding precisely so that you stop worrying about it.

Only 2,887 tokens were charged at full input price out of 112,617. That is the shape of a working cache: almost everything is a read.

Here is the failure that costs real money, and it is a one-line bug.

Put something that changes on every call near the front of the prompt — a timestamp, a request id, the user's name, a "today is" line, a freshly retrieved document — and the prefix differs from byte one. Nothing matches. Every call is a miss. And because every call presents a novel prefix, every call also writes.

Same conversation, same forty turns, caching enabled, with a per-call timestamp at the top of the system prompt:

Enabling prompt caching made the conversation twenty per cent more expensive than not enabling it. You paid the 1.25× write premium on 109,730 tokens and read back zero. There is no error, no warning, and the feature is switched on.

So the rule, and it is the whole of prompt caching in one line: stable content in front, variable content behind. System instructions, tool definitions and reference material first; timestamps, user identity and the current question last. Anthropic makes the hierarchy explicit — the cache follows tools → system → messages, and a change at any level invalidates that level and everything after it, so editing a single tool description invalidates the entire cache.⁵

Two consequences people trip over. Changing which tools are enabled changes the tool definitions, so a feature flag that adds a tool for some users splits your cache in two. And on Anthropic, toggling web search or citations modifies the system prompt, which invalidates the system and message caches without you touching a line of your own text.

The obvious response to a quadratic bill is to stop sending the whole history: keep the last dozen messages and drop the rest. It does reduce the bill, and it is usually the wrong move, and the measurement says why.

Truncating to a twelve-message window is 57 % cheaper than sending everything uncached — the comparison everyone makes, and why the technique is popular. But it is 39 % more expensive than sending everything with a working cache, and turning caching on alongside truncation makes it slightly worse rather than better.

The mechanism is the prefix again. A sliding window drops the oldest message every turn, so the prompt no longer starts where it started last time and every turn presents a new prefix. OpenAI's guidance says exactly this: "summarisation, compaction, or context truncation can change the prefix and reset cache reuse."⁶ By turn 40 the windowed prompt is 813 tokens, below the 1,024-token minimum, so it cannot be cached at all.

And the money is the cheap half of the cost. What you dropped is the instruction the user gave at turn 2 that the model needed at turn 40. Truncation trades a bill you can see for a failure you cannot, and doing it properly — compaction, structured notes held outside the window, retrieving history on demand — is Chapter 24's subject.

Long contexts are not merely more expensive because they are longer. Past a threshold they are more expensive per token, and the threshold applies retroactively to the entire prompt.

OpenAI's model page for gpt-5.6-terra states it in one sentence: "Prompts with >272K input tokens are priced at 2x input and 1.5x output for the full request."⁷ Not for the excess. For the whole thing.

prompt 271,999 + 500 output  ->  $0.5500
prompt 272,000 + 500 output  ->  $0.5500
prompt 272,001 + 500 output  ->  $1.0970

One token, fifty-five cents. If your service builds prompts from retrieved documents whose size you do not control, you have a cliff in your cost model at a boundary nobody on your team has written down.

Google's pricing works the same way with a 200,000-token threshold: Gemini 2.5 Pro is $1.25 per million input tokens for prompts up to 200K and $2.50 above it, with output going from $10.00 to $15.00.⁸ Anthropic went the other way — as of 6 September 2026 its documentation states that Claude 4.6 and later include the full one-million-token window at standard pricing, so "a 900k-token request is billed at the same per-token rate as a 9k-token request."⁹ Earlier models kept the surcharge.

This is why a price is not a number. A price is a table of tiers keyed by prompt length, which is what Tier[] in the cost function is for, and it is why computeCost selects the tier using the whole prompt rather than each bucket separately.

The five buckets map onto Chapter 13's two phases, and once you see the mapping the price ratios stop looking arbitrary.

Input tokens are prefill. The whole prompt goes through the model in one pass, processed in parallel — large matrix multiplications, compute-bound. Cost per token is low, and this is the phase that sets time to first token: a 4,947-token prompt has 4,947 tokens of prefill to do before the first word appears.

Output tokens are decode. They are produced one at a time, each a full forward pass that reads the entire KV cache, with the GPU mostly waiting on memory rather than computing. This is the phase that sets tokens per second, it cannot be parallelised within one response, and it is why output costs about six times input on the model priced here: $12.00 against $2.00 per million tokens.

Three consequences follow directly. A cache read replaces prefill work, so it buys latency and money at once — the same discount shows up as a lower bill and a shorter wait for the first token. Reasoning tokens are decode you never see, which is why a reasoning model streams nothing for several seconds and then answers quickly: Chapter 12 warned about the interface consequence, and this is the invoice consequence. And aborting a stream does not stop the generation — Chapter 14 built cancellation and left the price to this chapter, and the price is the full output count, because the tokens are produced and billed whether or not anybody is listening. The same is true of the answer nobody keeps: regenerating a turn 40 answer five times costs $0.057990 for the one left on screen.

The tokenizer of Chapter 7 was Python and stayed there. Budgeting happens in the server that builds the request, so it has to happen here, and there are exactly three levels of accuracy available.

Level one: count locally. js-tiktoken ships the same BPE merge tables as the Python tiktoken, so a byte-for-byte identical count for OpenAI encodings, with no network call:

import { getEncoding } from "js-tiktoken";

const enc = getEncoding("o200k_base");
const PER_MESSAGE = 4;   // role and delimiters added by the chat template
const PER_REPLY = 3;     // priming for the assistant turn

export function promptTokens(messages: { role: string; content: string }[]) {
  return messages.reduce(
    (sum, m) => sum + enc.encode(m.content).length + PER_MESSAGE, PER_REPLY);
}

The two constants matter and they are where local counts drift. Your text is not what gets tokenized — the chat template of Chapter 11 wraps every message in role markers first, and those are tokens you pay for. Four per message and three for the reply priming is the conventional approximation for OpenAI chat models; across the eighty-one messages of the conversation above they add up to 324 tokens, 6.4 % of its length. The counts here were cross-checked against the Python tiktoken from Chapter 7 on all eighty-one strings and are identical.

Level two: ask the provider. Anthropic exposes /v1/messages/count_tokens and Google exposes count_tokens, both accepting the same request shape as a real call and returning an input token count for free. Use them when you cannot count locally — and you cannot count locally for Anthropic, whose tokenizer is not published. Anthropic's documentation is careful about what it is giving you: the count "is an estimate", and it "may include tokens added automatically by Anthropic for system optimizations", for which "you are not billed".¹⁰

Level three: read usage in the response. That is the truth, and it arrives after the money is spent. Which is precisely why the first two levels exist — to decide whether to send the request, not to bill for it.

Four line items that do not appear as line items.

The system prompt, paid on every call. The one above is 192 tokens with its template overhead. Across forty calls that is 7,680 tokens — 5.6 % of this conversation's entire bill, for eight lines written once. It is also the best possible cache candidate, being both stable and first.

Tool definitions. Every tool's name, description and JSON schema goes out on every request, and providers add scaffolding on top. Anthropic publishes the number: enabling tools at all adds a hidden system prompt of 496 tokens on Claude Sonnet 4.5 with tool_choice set to auto, or 588 with any or a named tool.⁹ That is before your own schemas. Chapter 18 builds the catalogue; Chapter 24 measures what it eats.

Every generation, including the ones you discard. Five regenerations cost five times. The chat shows one.

Thoughts you are not shown. Billing is based on the full thought tokens though only a summary is returned, and no accounting of yours can audit that number.

One warning to close on, because it is the natural next thought and the answer is not the obvious one.

A million-token window does not mean a million usable tokens. Retrieval accuracy degrades with position: Liu et al. found that models locate information reliably at the beginning and the end of a long input and much less reliably in the middle.¹¹ A bigger window buys the ability to send more, not the certainty of being read.

That phenomenon is measured once in this course — the retrieval rate at nine positions in the same 853-token prompt — and it belongs in Chapter 24, where it changes what an agent does. It is cited here because it changes what you should buy: the cheapest token is the one you did not send.

You can now predict what a call will cost before you make it, read what it did cost afterwards, and tell the difference between the two. That covers everything about the request except the part you have not touched: the knobs.

Chapter 17 is sampling — temperature, top-p, top-k, the penalties, and the determinism you do not have. It starts by dismantling the most widespread error in the field, that temperature is a creativity dial. It is not: temperature divides the logits from Chapter 4 before the softmax, and raising it does not make the model imaginative, it raises the probability of tokens the model itself scored as worse. From there, why greedy decoding produces measurably worse text than sampling, why top-k and top-p fail on opposite shapes of distribution, and the experiment that ends the chapter: twenty identical forward passes at temperature 0 come back bit-for-bit identical when the model runs alone, and putting the same prompt in a batch alongside somebody else's requests moves 97 % of its logits.

They do not all match. The reason begins with the floating-point box from Chapter 2.

All prices, thresholds and multipliers in this chapter were read from the providers' own pages on 6 September 2026 and are stated with that date because they will change. The method matters more than the numbers: the buckets, the prefix rule and the tier arithmetic have been stable for two years while every figure in them has moved.

Stanford CS336 lecture 2, Resource accounting, is the closest academic treatment of this material and the right next read: it does the same arithmetic on the training side that this chapter does on the inference side. The token counts here were produced with js-tiktoken 1.0.21 using the o200k_base and cl100k_base encodings, over a forty-turn conversation of 5,090 tokens; the per-message template overhead is the conventional four-plus-three approximation and is stated wherever it is included. The cache, tier and truncation figures are the documented pricing rules applied to those measured token counts, not observations of live API responses — no paid call was made to produce this chapter, which is also the honest reason the latency claims are qualitative and the cost claims are not.