Tool Calling and Structured Outputs: The Contract That Holds

Before the mechanics, the sentence that prevents the most confusion: a tool call is a request, not an action.

The model emits a structured message that says I would like search_flights called with these arguments. Then it stops. Your code receives that message, decides whether to honour it, calls whatever it calls, and sends the result back as another message. The model never touched your database, never made an HTTP request, never had credentials.

Everything about agent security in Chapter 30 follows from that division, and so does everything about agent design in Chapter 23: the model proposes and your code disposes, and the code is where every guarantee lives.

So a tool, stripped of vocabulary, is two things:

None of this is emergent. As Chapter 11 established, tool calling is a trained behaviour:¹ during post-training the model saw thousands of conversations shaped exactly like this. That is why the format is model-specific, why reliability varies so much between models of similar size, and why a model can call a tool it has never seen — the shape was trained, the specific tool comes from your prompt.

Here is the tool as most people first write it. Note that nothing about it is wrong; it is just thin:

{
  name: "search_flights",
  description: "Search for flights.",
  parameters: {
    type: "object",
    properties: {
      from: { type: "string", description: "Airport." },   
      to:   { type: "string", description: "Airport." },   
      date: { type: "string", description: "The date." },  
    },
    required: ["from", "to", "date"],
  },
}

Twenty-four requests, six city pairs crossed with four ways of expressing a date ("the 3rd of next month", "next Friday", "15 December", "tomorrow"), greedy decoding so the results reproduce:

Read the first two columns before the last three. The model calls the right tool every time and produces well-formed JSON every time. The failure is entirely in the values, and the values are unusable: "Madrid" instead of MAD, "3rd October 2026" instead of 2026-10-03.

This is worth insisting on because it determines where you look when something breaks. The instinct is to add a JSON parser with a retry, or to ask the model more firmly for valid JSON. Neither addresses anything that happened here.

Same endpoint. Same code behind it. Same model, same prompts, same decoding. The only thing that changes is the text in the schema:

{
  name: "search_flights",
  description: "Search scheduled flights between two airports on a given day.",
  parameters: {
    type: "object",
    properties: {
      from: {
        type: "string",
        description: "Departure airport as a three-letter IATA code, e.g. MAD for Madrid. Never a city name.",   
        pattern: "^[A-Z]{3}$",
      },
      to: { /* same */ },
      date: {
        type: "string",
        description: "Departure date as an ISO 8601 calendar date, YYYY-MM-DD. Resolve relative dates against today before calling.",   
        format: "date",
        pattern: "^\\d{4}-\\d{2}-\\d{2}$",
      },
    },
    required: ["from", "to", "date"],
  },
}

The date format goes from 2 out of 24 to 24 out of 24. Perfect, from a text change, with no code touched and no retry logic. If you take one operational habit from this chapter, it is that: when a tool is called wrongly, the fix is almost always in the description, and it is the cheapest fix in the system.

Now read the second column, which is the more important half.

The date is in ISO format 24 times out of 24. It is the right day 12 times out of 24.

So half the calls now carry a perfectly formatted date that is the wrong date. The description told the model what shape to produce, and the model produced it flawlessly — but turning "next Friday" into 2026-09-11 requires knowing today's date and doing calendar arithmetic, and no amount of description supplies that. Same story for airports: format went from 4 to 16, but value only from 4 to 8, because writing MAD requires knowing that Madrid's airport is MAD.

That distinction is the load-bearing idea of the chapter:

A schema is a contract about form. It can make the model's output parseable, typed and consistent. It cannot make it true, and every failure mode that survives a good schema is a knowledge failure, not a format failure.

The two need different fixes, and confusing them wastes weeks. Format failures are fixed in the description or with constrained decoding, below. Knowledge failures are fixed by putting the knowledge in the prompt — the current date in the system message, an airport lookup as a second tool the model calls first, an enum in the schema when the set is small enough to enumerate. Note what all three have in common: they move the problem out of the model's memory and into its input, which is the whole of Chapter 24.

Everything above still relies on the model choosing to produce the right shape. There is a stronger guarantee available, and it is the best payoff of Chapter 17.

Recall how generation works: at every step the model produces a logit for every token in the vocabulary, and the sampler picks one. Constrained decoding inserts a step in between. Given a grammar — derived from your JSON Schema — it computes which tokens could legally come next, sets the logits of all the others to negative infinity, and lets the sampler choose from what remains.

If the schema says the next thing must be a {, then every token that is not { has probability zero. Not "unlikely": zero. The model cannot emit invalid JSON because the invalid tokens were removed from the distribution before sampling.

That is what "structured outputs", "JSON mode" and "guided generation" are underneath, and it explains their two properties. The guarantee is total for anything the grammar can express — types, required fields, enums, nesting — because it is enforced mechanically rather than requested politely. And it says nothing about content: a grammar can force "date" to be a string matching a date pattern, and cannot force it to be the right day. Which is the same wall as the previous section, arrived at from the other side.

Two practical notes. It is not free: the mask has to be computed at every step, and complex grammars cost measurable latency. And it changes what the model is doing — a model steered away from its preferred token can produce worse content while producing perfect structure, which is why "ask nicely and validate" is still a reasonable default for simple shapes and constrained decoding earns its cost when the shape is complex or the consumer is strict.

Chapter 14 measured a timeout followed by a retry billing two generations for one answer. With tools the same failure gets worse, because a tool can do something.

If your code calls charge_card, times out, and retries, you have two charges. The model has no idea any of this happened; it sees one tool result. The fix is the same as in any distributed system and it is not the model's problem: make the operation idempotent by giving the call a key, so the second execution recognises the first and returns its result instead of doing the work again.

The design rule that follows is worth stating plainly. Separate reads from writes in your tool catalogue. A read can be retried freely, run in parallel, and cached. A write cannot, and should carry a key, a permission check, and — for anything a user would want to know about before it happens — an approval step that puts a human between the request and the action. That approval step is not a courtesy: it is one of the few things standing between a prompt injection and a real consequence — and, Chapter 30 measures, the weakest of them.

The folklore says that loading many tools makes the model choose badly. It is worth measuring rather than repeating, so: the same twenty-four requests, with the flight tool plus a growing set of others — including three deliberately confusable ones (train timetables, ferry crossings, bus routes).

Selection did not degrade. With twenty tools, three of them plausibly confusable, a half-billion-parameter model picked the right one twenty-four times out of twenty-four. The dip at ten is three calls that named a different tool, and it does not survive going to twenty.

That is a negative result and it should be reported as one: on this task, with these tools, "too many tools" was not the problem. What did grow, monotonically and by a factor of six, is the prompt: 353 tokens to 2,119, paid on every request in the conversation, forever, whether or not any tool is used.

So the honest version of the folklore is about cost and context, not accuracy. Twenty tools is a permanent tax on every message, and Chapter 16 already showed what a permanent prefix does to a bill across forty turns. When people report that many tools hurt quality, the mechanism is usually that the definitions crowded out the context that mattered — which is a Chapter 24 problem wearing a Chapter 18 costume. Tools that are genuinely near-duplicates of each other are a real problem too, and the fix for those is not fewer tools but better descriptions and namespaces: prefix them by system (crm.search_customer, billing.search_customer) so that two catalogues merged from two teams do not collide, and so the model has something to discriminate on.

It helps to sort tools by what they do to the world, because the engineering differs for each.

Data tools read: search, fetch, query. Retryable, parallelisable, cacheable. They fail by returning nothing useful, and their main risk is that they bring untrusted text into the context — which is the entire attack surface of Chapter 30.

Action tools write: send, create, charge, delete. Not retryable without a key, not parallelisable safely, and the reason approval flows exist.

Orchestration tools call other models. A tool whose implementation is another agent, with its own prompt, its own tools and its own loop — and to the calling model it looks exactly like the other two, because a schema and an endpoint is all it ever sees.

That third kind is not a curiosity. It is the mechanism behind the agent-as-a-tool half of Chapter 25 — the other topology, the handoff, gives the conversation away and never gets it back — and it works precisely because the interface in this chapter is narrow enough that a whole agent fits behind it.

You now have a model that can ask for things, and a contract that makes the asking parseable. What you do not have is anything for it to ask about beyond what fits in its prompt.

The most common tool in production, by a wide margin, is a search over a body of text the model never saw during training: your documentation, your tickets, your contracts. That sounds like a solved problem — embed it, find the nearest neighbours, paste them in — and the parts that are not solved are the ones that decide whether the answer is trustworthy: how the text is cut up before it is embedded, what similarity threshold is low enough to mean I do not know, and how a citation gets attached to a claim so a reader can check it.

Chapter 19 is retrieval, and it is the chapter where a wrong answer stops being a curiosity and starts being a liability.

The measurements in this chapter come from Qwen/Qwen2.5-0.5B-Instruct with greedy decoding, over 24 generated requests crossing six city pairs with four date phrasings, using the model's own chat template for tool definitions. They reproduce exactly, and they are a small model: read the format/value split as a demonstration of the mechanism rather than as a benchmark of what current models do. A frontier model resolves "next Friday" correctly far more often — and still cannot be made to by a schema, which is the part that generalises.

The JSON Schema vocabulary used above (type, properties, required, pattern, format, enum) is specified in the JSON Schema draft that your provider's documentation names; the useful subset is small and the same across providers, and the differences that do exist — which keywords are enforced by constrained decoding rather than merely passed to the model — are worth reading in the provider's structured-output guide rather than assumed.

For constrained decoding as a technique, the guidance-style libraries and the outlines project document the grammar-to-logit-mask construction in a way that maps directly onto Chapter 17's sampler. And for the round trip itself, the clearest specification is not a tutorial but a protocol: Chapter 26 reads it line by line.