Your First Production LLM Call: Streaming, Retries, Timeouts

Read that table again and ask what kind of program it describes. It holds a connection open for forty seconds. It must be cancellable from a button. It accumulates a partial answer that is valid to display and invalid to save. And it runs in a server process or at an edge worker, next to the thing that renders the answer, holding a socket.

That is not a notebook. It is not that Python cannot do it — it can, and people do — it is that everything the previous thirteen chapters built was of a different kind. Chapters 1 to 13 held weights, gradients, logits and tokenizer bytes. From here the code holds a connection, a retry, a cancellation, accumulated state and, later, a permission prompt. The course changes language at exactly the seam where the object changes.

So the rule, written once:

If the code has weights, gradients, logits or tokenizer bytes in its hands, it is Python. If it holds a connection, retries, cancels, accumulates state and asks for permission, it is TypeScript.

The seam is single and it falls here, between Chapter 13 and Chapter 14. Three independent criteria put it here.

One: the ecosystem, counted. Everything the left half of this course cites is Python, and across the twelve courses audited for this syllabus there is not one precedent of backpropagation taught in another language: micrograd (17.4K stars), nanoGPT (62.8K), nanochat (57.8K), minbpe (10.7K), PyTorch (102.8K), transformers (164.9K). Writing Chapter 5 in TypeScript would break the link with those sources, and the links are half the value of a chapter that exists to be referenced rather than to rank. On this side the arithmetic reverses: Vercel's ai package is at 89.4M downloads a month and ships the thing itself — a tool-calling agent loop, exported as ToolLoopAgent — so the concept this course reaches in Chapter 23 has its reference implementation in TypeScript, even though, as that chapter measures, nobody has agreed on a name for it; Mastra is at 27.7K stars; and Anthropic's SDKs, generated from one specification, declare 202 endpoints in TypeScript against 201 in Python — parity, not a courtesy port.

Two: the normative source of MCP. The Model Context Protocol specification's schema is a schema.ts file. Teaching the protocol of Chapter 26 in another language means teaching a translation of its founding document.

Three: search demand, with a correction to the obvious guess. machine learning python is the most saturated phrase on the internet; ai agent typescript has its own healthy tail. But "the MCP ecosystem is mostly TypeScript" is only true depending on how you count: the official registry lists 8,275 servers on npm against 3,603 on PyPI, while by downloads Python wins — 287M a month for mcp plus 72M for fastmcp against 195M for @modelcontextprotocol/sdk. MCP is the one genuinely bilingual territory here, which is why Chapter 27 writes the same server twice instead of pretending.

You cannot learn any of this against a real provider. You cannot ask one for a 429 at a chosen moment, or for a socket that accepts your connection and never answers, or for a stream that stops in the middle of a word — and you would be paying for every experiment, when the interesting experiments are the ones you run a hundred times.

So the first program in this half of the course is not a client. It is a hostile server: forty lines of plain Node that speak the same wire protocol as a chat completions endpoint and misbehave on demand. Every number in this chapter came out of it.

import { createServer } from "node:http";

const WORDS = "A tide gauge is a device that measures sea level over time .".split(" ");
const CAPACITY = 3;                      // how many requests it will serve at once
let inflight = 0;

const sse = (res, obj) => res.write(`data: ${JSON.stringify(obj)}\n\n`);

createServer(async (req, res) => {
  const url = new URL(req.url, "http://x");

  if (url.pathname === "/hang") return;                        
  if (url.pathname === "/401") { res.writeHead(401); return res.end("{}"); }

  if (inflight >= CAPACITY) {                                  
    res.writeHead(429, { "retry-after": "1" });                
    return res.end(JSON.stringify({ error: { type: "rate_limit_error" } }));
  }
  inflight++;

  const cut = Number(url.searchParams.get("cut") ?? -1);       // abandon after N chunks
  const how = url.searchParams.get("how");                     // "close" = orderly, else reset
  const max = Number(url.searchParams.get("max_tokens") ?? 999);
  const delay = Number(url.searchParams.get("delay") ?? 60);   // ms per token

  res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
  for (let i = 0; i < Math.min(WORDS.length, max); i++) {
    if (i === cut) {                                           
      how === "close" ? res.end() : res.destroy();             
      inflight--; return;                                      
    }
    await new Promise((r) => setTimeout(r, delay));
    sse(res, { choices: [{ delta: { content: (i ? " " : "") + WORDS[i] }, finish_reason: null }] });
  }
  sse(res, { choices: [{ delta: {}, finish_reason: max < WORDS.length ? "length" : "stop" }] });
  res.write("data: [DONE]\n\n");
  inflight--;
  res.end();
}).listen(8787);

Four hostile behaviours, a line each: /hang accepts the socket and never writes to it; /401 refuses the key; the capacity check produces a genuine 429 with a genuine Retry-After header once three requests are already in flight; and ?cut=N abandons the answer halfway, either by resetting the socket or — with &how=close — by closing it in an orderly way, which turns out to matter a great deal. The rest is a real Server-Sent Events stream: one JSON object per data: line, a blank line between events, the string [DONE] at the end.¹

Run it, and the rest of the chapter is measurement.

node mock-provider.mjs &
curl -N "http://127.0.0.1:8787/v1/chat?max_tokens=3"

data: {"choices":[{"delta":{"content":"A"},"finish_reason":null}]}

data: {"choices":[{"delta":{"content":" tide"},"finish_reason":null}]}

data: {"choices":[{"delta":{"content":" gauge"},"finish_reason":null}]}

data: {"choices":[{"delta":{},"finish_reason":"length"}]}

data: [DONE]

A chat request is a list of messages, each with a role. That list is the model's entire state: there is no memory between calls, and whatever you want the model to know has to be inside the array you send this time. Chapter 15 is about what to put in it and Chapter 16 is about what it costs, so here it is just the shape.

const body = {
  model: "gpt-4.1-mini",
  messages: [
    { role: "system", content: "You explain instruments in one sentence." },
    { role: "user", content: "What is a tide gauge?" },
  ],
  stream: true,
  max_tokens: 200,
};

Those roles are not decoration. They are rendered into the chat template of Chapter 11 before the model sees a single token, which is why sending the wrong role silently degrades the answer instead of raising an error.

One rule with no exceptions: the API key never travels to the client. Not in an environment variable prefixed for the browser, not in a build-time constant, not "temporarily". A key in a bundle is a key on someone else's bill within days. The browser talks to your server, your server holds the key and talks to the provider — and because your server is in the middle, it is also the only place that can meter what each user spends, which is where the accounting of Chapter 16 has to live.

Now the experiment the chapter is built on. One question, one mock provider producing thirteen tokens at 60 ms each, three ways of asking.

First, without streaming. The client sends the request and waits for the whole JSON body.

blocking   first visible =  791 ms   complete =  791 ms   finish_reason = stop

The two numbers are the same, and that is the entire problem. For 791 ms the user has a spinner, and not one word was available earlier — the server had the answer, byte by byte, and chose to say nothing.

Second, with streaming. Same server, same answer, same total work. The difference is a parser.

export async function* readSSE(res: Response) {
  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });   
    let sep: number;
    while ((sep = buffer.indexOf("\n\n")) !== -1) {       
      const event = buffer.slice(0, sep);
      buffer = buffer.slice(sep + 2);
      for (const line of event.split("\n")) {
        if (!line.startsWith("data:")) continue;
        const payload = line.slice(5).trim();
        if (payload === "[DONE]") return;
        yield JSON.parse(payload);
      }
    }
  }
}

Three details there are load-bearing and most first attempts skip all three. The buffer exists because a network chunk has no relationship to an event: one read() can return half an event, or two and a half. The { stream: true } flag exists because a multi-byte UTF-8 character can be split across two chunks, and without it an accented letter becomes a replacement character at random. And events are separated by a blank line, not a newline, which is why the loop looks for \n\n.

streaming  first visible =   65 ms   complete =  793 ms   finish_reason = stop

Twelve times faster to the first word, and two milliseconds slower to the last. Streaming makes nothing faster. It changes what the user is doing during the same 790 ms: reading instead of waiting. That is the whole benefit, it is enormous, and it is the reason every chat product streams.

Third, with twenty clients at once. The mock provider serves three requests at a time. Fire twenty:

jitter=true  clients=20  server capacity=3
HTTP requests made: 74   429s received: 54   200s: 20
wall clock: 7,100 ms
retries per client: 0 0 0 1 1 1 2 2 2 3 4 3 3 5 4 5 4 5 4 5
every answer identical: true

Twenty answers, seventy-four requests, fifty-four rejections. Nobody lost anything, every client got the same text, and the only visible cost was time. That is a retry policy working. The rest of this chapter is about the three ways it can fail instead.

Before the failures, the field almost everybody ignores on the first pass. Every stream ends with an event carrying finish_reason. stop means the model decided it was done. length means it hit the token ceiling, so the answer is truncated mid-sentence and it is not the model's fault. Later chapters add tool_calls (Chapter 18) and content filters.

Now watch two endings a naive client cannot tell apart. Same server, same delay, one truncated by max_tokens and one where the connection is closed cleanly after five tokens:

max_tokens=5           loop ended NORMALLY   chunks=5  finish_reason=length  text="A tide gauge is a"
socket closed cleanly  loop ended NORMALLY   chunks=5  finish_reason=null    text="A tide gauge is a"
socket destroyed       threw TypeError: terminated (UND_ERR_SOCKET)
                                             chunks=4  finish_reason=null    text="A tide gauge is"

Read the first two rows carefully. Identical text. Identical chunk count. No exception in either case. The for await loop finished normally both times, because from the reader's point of view the body ended and that is all a body can do. The only difference in the entire observation is that one carries finish_reason: "length" and the other carries nothing at all.

So the rule is not "catch errors while streaming". It is:

A stream that ends without a finish_reason did not end. It stopped.

Treat a missing finish_reason as a failure, always, and never persist that text as a completed answer. The third row shows the easier case — a destroyed socket does throw, and it also loses the chunk that was in flight, which is why the text is one word shorter than the two above.

The most expensive habit a new product has is one catch block for everything the provider returns. These codes are not variations of "it failed". They are five instructions, and four of them contradict each other.

The line that matters runs between 4xx and the rest. A 400 or a 401 returns exactly the same answer if you send it a thousand times, because nothing on either end changes between attempts. Retrying it is not caution, it is a delay with extra steps. Measured: one client that makes six attempts — five retries with exponential backoff —, and one that reads the code first.

retry everything  ->  6 requests, gave up after 6,325 ms, still HTTP 401
triage first      ->  1 request,  gave up after     4 ms, still HTTP 401

Six seconds of spinner to reach an answer that was available in four milliseconds. And that is the mild version: retries in a product are usually nested — a retrying HTTP client inside a retrying job runner inside a queue with its own redelivery — so six seconds becomes six minutes of a permanently broken deployment looking like a slow one.

The triage is nine lines and belongs in one place:

export type Verdict = "retry" | "retry-after" | "fatal";

export function classify(status: number): Verdict {
  if (status === 429) return "retry-after";        
  if (status === 408 || status >= 500) return "retry";
  return "fatal";  // 400, 401, 403, 404, 422 — nothing changes by waiting
}

Two more for your list: 402, which some providers use for "you are out of credit" and which needs a screen with a link to buy more rather than a retry, and 529 or its vendor-specific equivalents, which behave like 503.

Retrying is easy. Retrying when is the part with a measurable right answer.

Exponential backoff is the standard: wait a base delay, double it after each failure, stop at a ceiling. It exists because an overloaded server gets worse if the clients that just failed come straight back.

The problem is that everyone doubles from the same starting point. If a hundred clients hit a limit at the same moment — and they will, because that is what a traffic spike is — then all hundred wait 200 ms, all hundred retry together, all hundred fail together, and all hundred wait 400 ms. The retry schedule has synchronised them. That is a thundering herd, and randomness is the fix.²

That single change — picking uniformly from the interval instead of taking its upper end — is called full jitter. It is one call to Math.random(), and it is worth measuring rather than believing:

export const backoffNaive = (n: number, base = 200, cap = 20_000) =>
  Math.min(cap, base * 2 ** n);

export const backoffFull = (n: number, base = 200, cap = 20_000) =>
  Math.random() * Math.min(cap, base * 2 ** n);      

A hundred clients, one server that serves three at a time, everything else identical, three runs each:

Two things in that table, and the second is the important one.

The first is the median: 226 seconds against 2.2, a factor of about a hundred, with less than half the requests. The busiest retry window says why. Without jitter, up to 97 of the hundred clients arrived inside the same 50-millisecond slot; the server had three, so 94 were rejected and went to sleep together, still synchronised, to do it again with a longer wait. With jitter the same hundred spread across the same windows in groups of around thirty and drained almost immediately.

The second is the variance. Without jitter: 65.6 s, 245.7 s, 225.6 s. With it: 2.2, 2.3, 1.8. A system without jitter does not merely perform badly, it performs unpredictably, because the outcome is decided by microscopic scheduling accidents that pick which three of a hundred synchronised clients arrive first. That is the signature of this bug in production: an endpoint that is fine, fine, fine, and then takes four minutes, and no change of yours explains it.

And the cheapest retry is the one that never happens. Put a concurrency gate in front of the provider — a counter that never lets more than N requests be in flight — and the same twenty clients that needed 74 requests and 7.1 seconds behave like this:

client-side gate of 3: 20 HTTP requests, 0 429s, wall 883 ms

Twenty requests for twenty answers, zero rejections, eight times faster. A retry is the apology; the gate is not needing one.

When a provider returns 429 it usually tells you how long to wait, in the Retry-After header.³ That number is not advice: the provider is the only party in the exchange that knows when its window resets.

So the wait is the larger of the two: never less than Retry-After, and never less than your own backoff either, because the header tells you when the limiter forgives you and not when the server has room.

const header = res.headers.get("retry-after");
const floor = header ? Number(header) * 1000 : 0;   // seconds -> ms
const wait = Math.max(floor, backoffFull(attempt));  

The trace of the unluckiest client in the twenty-client run shows the header doing its job. Its first four backoff draws were all below one second, and all four were overridden:

t+   26ms  attempt 0  HTTP 429  -> sleep 1000 ms
t+ 1032ms  attempt 1  HTTP 429  -> sleep 1000 ms
t+ 2034ms  attempt 2  HTTP 429  -> sleep 1000 ms
t+ 3046ms  attempt 3  HTTP 429  -> sleep 1000 ms
t+ 4047ms  attempt 4  HTTP 429  -> sleep 2782 ms
t+ 6852ms  attempt 5  HTTP 200  -> sleep 0 ms

Two practical notes. Retry-After may be an HTTP date rather than a number of seconds, so parse both. And providers rate-limit on two axes at once — requests per minute and tokens per minute — which is why long prompts get rejected far below the documented request limit. The header looks the same in both cases; the fix is not.

Ask the mock provider for /hang. It accepts the connection, and then does nothing at all: no headers, no body, no close. This is not exotic — it is what a load balancer does when the process behind it has died without closing its sockets.

Two clients, one difference:

AbortSignal.timeout(5s)   gave up after   5.0 s  (TimeoutError: The operation was aborted due to timeout)
no timeout                gave up after 300.8 s  (TypeError: fetch failed)
                          cause: HeadersTimeoutError UND_ERR_HEADERS_TIMEOUT

Three hundred seconds. Five minutes of a socket held open, a request slot occupied and a user staring at a spinner, ending in a generic TypeError that says nothing about what happened. That number is not a bug: it is Node's default headers timeout, reasonable for a generic HTTP client and catastrophic for a user-facing request. Every runtime has such a default, most people never look it up, and the only way to find yours is to hang a socket on purpose the way we just did.

So: every outgoing request gets an explicit deadline, chosen by you.

const res = await fetch(url, {
  method: "POST",
  headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
  body: JSON.stringify(payload),
  signal: AbortSignal.timeout(20_000),   
});

For a streaming call one deadline is not enough, because there are two different failures. The first is the stream never opens: no event arrives at all, and ten to thirty seconds is right. The second is the stream opens and then stalls: tokens flowed and then stopped, forever, with the socket still healthy. A total-duration timeout cannot tell a stalled stream from a long correct answer, so what you want is an idle timeout — a timer reset by every event, firing only when nothing has arrived for, say, fifteen seconds.

Cancellation is the same machinery pointed at a person. AbortSignal.timeout and a user pressing Stop both arrive as an AbortError, so combine them and record which one fired:

const user = new AbortController();
const signal = AbortSignal.any([user.signal, AbortSignal.timeout(20_000)]);
// stopButton.onclick = () => user.abort();

Aborting matters for a reason beyond tidiness: the tokens are being generated and billed while you are not listening. Chapter 16 puts a price on that.

Now the failure that costs money rather than time. A request times out on the client, and the obvious move is to send it again — but a timeout tells you nothing about whether the server received it. Very often it did, and is still working.

Measured. The mock provider needs 780 ms for the answer. The client gives up at 300 ms and retries. The server counts how many answers it actually generated, which is what it would bill:

idempotency-key: no    attempt 0: TimeoutError after 300 ms  |  attempt 1: TimeoutError after 300 ms
                       answers generated (and billed): 2

idempotency-key: yes   attempt 0: TimeoutError after 300 ms  |  attempt 1: HTTP 200 (replay) id=cmpl_1
                       answers generated (and billed): 1

Without a key: two full generations, paid for twice, and the client received neither of them. With a key: the server recognised the second request as the same request and replied instantly with the answer it had already produced, so the retry both avoided the double charge and was the attempt that finally succeeded.

An idempotency key is a unique string you generate per logical operation — not per attempt — and send unchanged on every retry of it. The server stores the outcome against the key and replays it. It is the mechanism payment APIs use, for the same reason.⁴

async function send(url: string, payload: unknown) {
  const key = crypto.randomUUID();      // once per turn, not per attempt

  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(url, {
      method: "POST",
      body: JSON.stringify(payload),
      headers: { "content-type": "application/json", "idempotency-key": key },  
      signal: AbortSignal.timeout(20_000),
    });
    if (res.ok) return res;
    if (classify(res.status) === "fatal") throw new Error(`HTTP ${res.status}`);
    await sleep(backoffFull(attempt));
  }
  throw new Error("out of attempts");
}

Two honest limits. Not every provider supports idempotency keys on completions, and where the endpoint is not idempotent, the correct number of retries for a POST that may already have run is zero. And a stream that failed halfway is not replayable in the general case: you either restart it and pay again, or keep the partial text and mark it incomplete. Which of those your product does is a product decision, not a networking one, and it is worth making on purpose.

The client written in this chapter has no idea what is behind the port. Point its base URL at a commercial provider and it streams tokens from a model of a trillion parameters. Point it at a server built on Chapter 13's arithmetic — serving the model you pretrained in Chapter 10, with its KV cache and its quantized weights — and the same code, unchanged, streams tokens from a model you built.

const BASE = process.env.LLM_BASE_URL ?? "http://127.0.0.1:8000/v1";  

That single line is the seam of this course. On one side of it is what the first thirteen chapters built; on the other, what the next sixteen build. The boundary is clean because the contract is HTTP and SSE, and neither side knows anything else about the other.

It is worth noticing what you lost by crossing. Behind a commercial endpoint you control neither the weights, nor the sampling implementation, nor the version you are talking to, nor whether it changed this morning. What you control is the contract: the messages you send, the deadline you set, the codes you distinguish, and what you do when nothing comes back. That is a smaller surface than you had in Chapter 5, and every remaining chapter is about using it well.

You now have a client that streams, gives up on time, retries the right things and never retries the wrong ones. What it sends is still whatever you typed.

Chapter 15 is about that content, and it comes with a discipline. The internet is full of prompting advice — offer the model a tip, threaten it, tell it to take a deep breath — and almost none of it arrives with a measurement. Some of those techniques move the output a great deal, some move it not at all, and at least one makes a classification task worse while costing more tokens. Which is which is not obvious from reading them, and it is not settled by argument.

So the next chapter builds a bench: sixty cases with known answers, four variants of the same prompt, run in parallel through exactly the client you just wrote, tabulated with the confidence intervals from Chapter 4 — because four variants over twenty cases distinguish nothing at all. One sentence governs the whole chapter: a prompt is measured, not debated.

Every number above came from the mock provider, on Node 22 over a loopback interface, so the latencies are cleaner than any real network will give you. That is deliberate: none of the failures being measured is caused by the network, and a hostile server you can restart teaches better than a real one you must pay for and cannot break.