Fine-Tune, Retrieve or Prompt? The Decision Is Economic

"Should we fine-tune?" is asked as if it were a question about a model. It is a question about a budget, with a shape no benchmark answers: what is paid once, what is paid per question, and what is paid again every time the world moves.

The three routes are also not three ways of doing one thing, and the vendors say so more plainly than most blog posts. OpenAI's own table of what supervised fine-tuning is best for lists four uses: classification, nuanced translation, generating content in a specific format, and correcting instruction-following failures.¹ None of them is "teaching the model something it does not know". Its summary of the benefit is that "you can use shorter prompts with fewer examples and context data, which saves on token costs at scale and can be lower latency" — an argument about the invoice, from the company selling the feature.

So:

• Fine-tuning teaches form and behaviour. Tone, format, the shape of an answer, a boundary you can demonstrate but not describe. The strongest published version is LIMA's Superficial Alignment Hypothesis: knowledge comes from pretraining, alignment mostly teaches which sub-distribution of formats to speak in — which is why a thousand curated examples sufficed there.²
• Retrieval supplies facts that change. It is the only one of the three where an edit to your documentation reaches the answer without touching the model.
• Prompting covers most real cases, and is the honest baseline. In-context learning has been the default since Language Models are Few-Shot Learners: the task is demonstrated inside the prompt and no weight moves.³

Two measured papers close the door on the mistake in the middle. Ovadia and colleagues compared injecting knowledge by unsupervised fine-tuning against injecting it by retrieval, and retrieval won consistently, including on facts the base model had already seen in pretraining.⁴ Gekhman and colleagues measured the damage: examples that introduce new knowledge are fitted slowly, and as the model finally fits them its hallucination rate on other questions rises.⁵ Teaching facts by fine-tuning does not merely fail; it degrades answers you were not training on.

That half is settled. The economic half is not, and it is the rest of the chapter.

One case, run three ways: technical support over your own documentation, which changes every week.

The corpus is real and on this disk: the 23 Markdown documents that a working software repository keeps as internal documentation — the build guide, the brand rules, the translation brief, ten service manuals, the performance and security notes. Measured with o200k_base, the encoding from Chapter 7:

documents                              23
characters                        159,223
words                              22,194
tokens (o200k_base)                42,921
tokens with per-file headers       43,158

Forty-three thousand tokens is a comfortable size for this decision: it fits any modern window, so all three routes are genuinely available. At ten million the decision is made for you, and it is retrieval.

Now the work the word "weekly" is doing. Documentation churn is usually asserted; here it is counted, from that repository's version history:

The documents move about every other week. The user-visible strings — which are what a support desk is actually asked about — moved in every week they have existed, at about twenty commits a week. Whatever route we pick has to survive that, and "how often does the thing you trained on change?" turns out to have a number in your own repository rather than an opinion.

Twenty realistic support questions were written against this corpus, one per topic, and every figure below is computed over those twenty.

The simplest thing that works: put the whole corpus in the system prompt, the question at the end, and let the model find it.

system instructions                       140 tokens
the 23 documents                       43,158 tokens
the question (median of 20 measured)       13 tokens
the answer (the one assumption)           150 tokens

Every number there was counted except the last: 150 output tokens is an assumption, chosen inside the range of assistant turns Chapter 16 billed. It is the only figure here that was not executed, it is applied identically to all three routes, and the break-even section shows exactly how much the conclusion moves when you change it.

At the rates read from the provider's page on 7 September 2026 — $1.50 per million input tokens, $9.00 per million output⁶ — that is $0.066317 a question. You are paying to re-read forty-three thousand tokens in order to answer thirteen.

Chapter 16's fix applies directly: the corpus is stable and it is at the front, so it is a perfect cache prefix, and reading it back costs a tenth — $0.007864 a question, an 88 % cut. Chapter 16's warning applies too, in the form that chapter flagged and did not price. This provider charges no write premium; it charges rent. An explicit cache costs $0.000001 per stored token per hour,⁶ so keeping 43,298 tokens warm costs

whether or not anybody asks anything. That is $189.78 over six months, for an empty room. Divide the rent by the saving per question and the condition comes out in one line: caching this corpus pays for itself above 0.74 questions an hour — 546 a month once the weekly cache rebuild is counted too. Below that, the feature you enabled to save money loses it.

Same route, same code, one flag, five times the bill. Chapter 16 found a version of this caused by a timestamp in the wrong place; here nothing is wrong except the traffic. A cache is a bet on volume, and on this provider you place it by the hour.

Chapter 19's retriever, unchanged: cut on section boundaries with a contextual header, index, put the four best extracts in the prompt. Measured over the twenty questions:

chunks produced from the corpus              330
mean tokens of a chunk's own text          124.9
mean tokens of the four retrieved extracts   884
prompt per question (140 + 884 + 13)       1,037
one-off embedding of every chunk        46,823 tokens

Forty-two times fewer prompt tokens than route one, at $0.002906 a question. The index costs $0.0070 to build at $0.15 per million embedding tokens⁶ — less than three questions' worth — and the same $0.0070 to rebuild from scratch every time the documentation changes. Rebuilding the entire index every week for six months costs eighteen cents.

One thing is worth stopping on. Retrieval destroys prompt caching. The stable prefix is now the 140-token system instruction; from token 141 the prompt differs on every call, because the extracts are chosen per question. And 140 tokens is below every cache minimum Chapter 16 quoted. So route two cannot be cached at all, which sounds bad and is not: not caching 1,037 tokens is cheaper than caching 43,298.

That is a general rule worth carrying: the two big token-saving techniques are mutually exclusive on the same content, and the one that wins is whichever removes more tokens. Retrieval removes 97.6 % of them.

Train on two hundred examples in the house style, then ask questions with no documentation attached at all.

training examples                            200
training tokens                           24,389
epochs                                         3
prompt per question (15 + 13)                 28

Training costs 24,389 × 3 × $10.00 per million = $0.7317. That is the entire construction cost, less than a cup of coffee, which is exactly why so many teams pay it before checking whether it helps.

Now the trap, and it is the reason this chapter exists. A fine-tuned model does not cost the same as its base model to run. The pricing page says it in one sentence: "for model inference starting from Gemini 3, tuned model endpoint prediction price will be 1.5 times of the base model."⁶ Not the training. The inference, on every token, for as long as the model lives.

So put it in a formula. Let $p_i$ and $p_o$ be the base input and output prices, $m$ the tuned multiplier, $L_R$ the prompt length of the route you are replacing, $L_F$ the prompt length after fine-tuning, and $O$ the answer length. Fine-tuning is cheaper per question only when

The first term is obvious: your new short prompt, marked up. The second is not, and it is where the money goes — the surcharge on the answer, which has nothing to do with your prompt and which training cannot shorten. With the measured numbers — $m = 1.5$, $L_F = 28$, $O = 150$, $p_o/p_i = 6$ — the threshold is

answer   50 tokens -> the prompt it replaces must exceed   192 tokens
answer  150 tokens -> the prompt it replaces must exceed   492 tokens
answer  400 tokens -> the prompt it replaces must exceed 1,242 tokens
answer 1000 tokens -> the prompt it replaces must exceed 3,042 tokens

At the measured answer length, 492 tokens — of which 450 are the answer surcharge, not the prompt. Replacing a shorter prompt than that is more expensive per question, forever, at any volume; and the threshold grows linearly with how much your assistant says, so one that writes long answers can never fine-tune its way to a cheaper token however much prompt it deletes.

The same fact from the other end is the sentence to remember. Of the fine-tuned route's $0.002088 per question, 97.0 % is the answer. Fine-tuning optimises the remaining three per cent.

Four numbers describe any of these routes: what you pay once, what you pay when the documentation changes, what you pay hourly regardless, and what you pay per question. That extends Chapter 16's computeCost without modifying it.

import { computeCost, type Pricing, type Usage } from "./cost";   // Chapter 16

export interface Route {
  name: string;
  setupUSD: number;            // paid once, before the first question
  perRefreshUSD: number;       // paid every time the documentation changes
  standingUSDPerHour: number;  // paid per hour whatever the traffic
  pricing: Pricing;
  usage: Usage;                // one question and its answer
}

export const perQueryUSD = (r: Route) => computeCost(r.pricing, r.usage);

const HOURS_PER_MONTH = (24 * 365.25) / 12;

export function totalUSD(
  r: Route, months: number, queriesPerMonth: number, refreshesPerMonth: number,
) {
  return r.setupUSD
       + months * refreshesPerMonth * r.perRefreshUSD
       + months * HOURS_PER_MONTH * r.standingUSDPerHour
       + months * queriesPerMonth * perQueryUSD(r);
}

/** Monthly volume at which `b` overtakes `a`. null = it never does. */
export function crossover(
  a: Route, b: Route, months: number, refreshesPerMonth: number,
): number | null {
  const fixed = (r: Route) =>
      r.setupUSD
    + months * refreshesPerMonth * r.perRefreshUSD
    + months * HOURS_PER_MONTH * r.standingUSDPerHour;
  const dFixed = fixed(b) - fixed(a);                     // b's extra fixed cost
  const dVar = perQueryUSD(a) - perQueryUSD(b);           // b's per-question saving
  if (dVar <= 0) return null;                             // b is never cheaper
  return Math.max(0, dFixed / dVar / months);
}

The tuned model is not a different price list, it is the same one multiplied:

const TUNED_MULTIPLIER = 1.5;   // read from the provider's pricing page, 2026-09-07

const scale = (p: Pricing, k: number): Pricing => ({
  input: p.input.map(t => ({ ...t, price: t.price * k })),
  cachedInput: p.cachedInput!.map(t => ({ ...t, price: t.price * k })),
  output: p.output.map(t => ({ ...t, price: t.price * k })),   
});

That one highlighted line is the previous section's whole argument written as code: the multiplier lands on output too.

Six months, with the documentation refreshed weekly:

And the crossovers, which are the four numbers a budget actually needs:

retrieval -> fine-tune, documentation never changes:     148 questions / month
retrieval -> fine-tune, documentation refreshed weekly: 3,989 questions / month
prompt (no cache) -> retrieval:                            1 question / month
prompt (no cache) -> prompt (cached):                    546 questions / month

Read the first two together, because they are the point of the chapter. A stationary corpus makes fine-tuning pay for itself in a hundred and fifty questions; a corpus that changes weekly moves the same crossover by a factor of twenty-seven, and nothing about the model changed — only how often you pay for it again. Construction cost is a footnote; maintenance cost is the decision.

If you now conclude that a busy support desk should fine-tune, the arithmetic agrees with you. It is still wrong, and the next section is why.

The cost sheet has one column it cannot compute, so this section runs the fine-tune: locally, on a small open model, with the adapter written by hand rather than pulled from a library. Chapter 11 built LoRA; here it is, on the q_proj and v_proj of all 24 layers of Qwen2.5-0.5B-Instruct at rank 8:

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__(); self.base = base
        for p in self.base.parameters():
            p.requires_grad = False              # the model is frozen  
        self.A = nn.Parameter(torch.zeros(r, base.in_features))
        nn.init.normal_(self.A, std=1 / r)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.s = alpha / r
        self.on = True                           # so the same run can compare both

    def forward(self, x):
        y = self.base(x)
        return y + (x @ self.A.T @ self.B.T) * self.s if self.on else y

The two hundred training examples come mechanically from the corpus, so they reproduce: the question is a section heading turned into a question, the answer is that section's own text in a rigid house style — one line beginning Short answer:, one line beginning Source: with the file path. The format is the form being taught; the path is the fact. Then two numbers over twenty held-out questions: does the answer come out in the house style, and does it name the file that really answers the question?

Two baselines make the table readable, and both are Chapter 4's insistence rather than an afterthought. Ten of the twenty right answers are the same file, so a model that ignores the question and always answers CLAUDE.md scores 10/20. And the retriever has its own ceiling: across these twenty questions its four extracts contain the right file 14 times and rank it first 7, so 14/20 is the most any reader could score using it.

LoRA modules 48   trainable parameters 540,672 (0.109 % of the model)
400 steps, 2 epochs, 0.76 s/step on 16 CPU threads, 304 s in total
mean loss over the first 50 steps 3.7363 -> over the last 50 steps 2.4197

                                        house style   correct source
always answer the most common file             --          10 / 20
the retriever's own ceiling                    --          14 / 20
base model, closed book                    0 / 20           0 / 20
fine-tuned, closed book                   19 / 20           8 / 20
base model, four retrieved extracts       13 / 20           2 / 20
fine-tuned, four retrieved extracts        1 / 20           1 / 20

The form was learned, completely and fast. Zero to nineteen out of twenty, from a 540,672-parameter adapter — 0.109 % of the model — in five minutes of training on a processor with no graphics card in sight.

The facts were not. Eight out of twenty is not distinguishable from the ten you get by ignoring the question entirely, and Chapter 4's interval on twenty samples says so out loud. Those file paths were in the training data three times over; what came out was the habit of ending with a plausible-looking Source: line. Asked the question at the top of this chapter, the fine-tuned model answered Short answer: 10.x . . . and cited CLAUDE.md. The right answer, which is in CLAUDE.md, is 18.17.0.

And then the form broke, which is the row that justifies the experiment. Hand the fine-tuned model a thousand tokens of retrieved extracts — a prompt shape it never saw, since every training prompt was twenty-eight tokens — and the house style collapses from 19/20 to 1/20. On the question at the top of this chapter it answers 18.17.0 — correct, and with none of the format it was trained for. So fine-tuning did not teach a format; it taught a format conditional on the prompts in the training set, and the first prompt that looked different took the format with it. Whatever you fine-tune on becomes the one input distribution your model is good at, and nobody puts that in the spreadsheet.

A last note about the metric, pointing straight at Chapter 29: "correct source" scores form and fact together, which is why both retrieval rows look terrible even though both models got that question's fact right. One end-to-end number was hiding three things — a retriever at 14/20 recall, a 0.5B reader and a citation format — and choosing which to fix means separating them before you measure, not after.

Now the column the vendors fill in for you. A fine-tuned model is not an asset you own; it is a lease on somebody else's base model, with an end date printed on it. On 7 September 2026 the fine-tuning section of OpenAI's pricing page carried this notice in full:

OpenAI is winding down the fine-tuning platform. The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months. All fine-tuned models will remain available for inference until their base models are deprecated.⁷

The timeline is dated to the day: 7 May 2026, closed to organisations that had never fine-tuned; 2 July 2026, closed to those that had not run inference on a fine-tuned model in sixty days; 6 January 2027, no new jobs at all.⁸ The same page schedules the shutdown of the fine-tuned models themselves — ft-gpt-3.5-turbo, ft-gpt-4, ft-gpt-4.1-nano, ft-babbage-002, ft-davinci-002 — on 23 October 2026, each with a recommended replacement base model, which is a polite way of saying: train it again.

The other frontier vendor never sold you the lease. Anthropic's documentation index lists 699 pages and not one is about fine-tuning; the model-customisation sections of the Bedrock pricing page cover Amazon Nova, Amazon Titan, Cohere, Meta and OpenAI open-weight models, and no Claude.⁹¹⁰ If your architecture depends on a fine-tune, one of the three frontier families is simply unavailable to you at any budget.

Self-hosting replaces the lease on a model with a lease on a machine, and AWS does that arithmetic on its own page: one model unit of provisioned throughput for a customised model, one-month commitment, is "1 model unit × $21.18 × 24 hours × 31 days = $15,757.92" a month.¹⁰ Renting the metal directly is cheaper and not free — $3.99 per GPU-hour on demand for an H100, $1.99 preemptible¹¹ — roughly $2,900 a month for one card that has to be up whether or not anyone asks anything. The whole retrieval route at ten thousand questions a month is $174.52 for six months.

This is where LoRA earns its place, as a budget argument rather than a technical one. Measured on the same model, a rank-16 adapter over attention and the feed-forward layers is 8,798,208 parameters — 1.781 % of the model, 17.6 MB in bfloat16 — against 0.988 GB of base weights, and its optimiser and gradient state is 140.77 MB where full fine-tuning needs 7.90 GB, a factor of 56. The consequence is not cheaper training but that one loaded base model can serve many adapters, which is the only way a GPU's fixed cost gets divided by anything. Managed training reflects it: $0.48 per million tokens low-rank up to 16B against $0.54 full, with a $4.00 minimum per job.¹¹ That floor is the detail. At 24,389 tokens for three epochs, every retraining on this corpus bills $4.00 rather than the $0.04 it computes to — $104 of minimums across twenty-six weekly runs, for ninety-one cents of arithmetic.

Two more columns that only appear on the invoice.

Data residency costs about ten per cent, and two providers agree on the figure. OpenAI charges "a 10 % uplift" on data-residency endpoints for models released on or after 5 March 2026;⁷ Vertex prices its non-global endpoints at $1.65 against $1.50, the same ten per cent.⁶ Set that against the fifty per cent a tuned endpoint costs and the folklore inverts: residency is cheap and fine-tuning is not — and fine-tuning is not the private option anyway, since the corpus reaches the provider either way, once at training time instead of once per call.

The most explicit price ever put on your data is on the same page, which lists one fine-tuned model twice: with data sharing enabled, inference is exactly half — $2.00 against $4.00 input, $8.00 against $16.00 output.⁷ Letting the provider keep what you sent is worth a 50 % discount, which tells you what it is worth to them.

Distillation — training a small model of your own on a large one's answers — is usually offered as a way out of both. Price it and it is not, because the teacher is the system you were trying to replace: producing two hundred training examples by asking the retrieval route two hundred questions costs 200 × $0.002906 = $0.58, on top of the $0.73 to train on them. Distillation is something you do after the retrieval pipeline works, to make it cheaper, and it inherits every fact the retriever got wrong.

Money is the visible half. The other arrives as a wait, with the same cause as the bill: the model reads the whole prompt before it says a word. Chapter 13 measured prefill against decode on a model you could touch; here is the same measurement, one run, one machine, against prompt length:

The absolute numbers belong to a 0.5B model on sixteen CPU threads and say nothing about a hosted frontier model. The shape transfers exactly: prefill grows with prompt length, and the cost per token creeps up as Chapter 9's quadratic term starts to show — 4.79 ms at a thousand tokens against 6.04 ms at eight thousand, a 26 % penalty for merely being longer.

The consequence for the three routes is direct. Route one prefills forty-three thousand tokens per question, and a cache hit is what makes that bearable — Chapter 16 explained why: a cache read replaces prefill work, so it buys latency and money in one transaction. Route two prefills a thousand and adds a round trip to the index first. Route three prefills twenty-eight and adds nothing, which makes it measurably the fastest of the three at answering. It is just answering the wrong thing.

Three failures that look like model problems and are not — ten minutes here saves a month later:

And the requirement over all of it: this decision cannot be made without an evaluation set, and the vendor selling the fine-tune says so. OpenAI's guide opens with "Only invest in fine-tuning after setting up evals. You need a reliable way to determine whether your fine-tuned model is performing better than a base model", and adds that if fifty good examples change nothing, the problem is the task or the prompt, not the data volume.¹ Twenty questions, which is what this chapter used, shows a mechanism and cannot choose a supplier — Chapter 4 measured why, and what to do when twenty cases are all you have — repeat them, pair them, and measure the spread between runs — is Chapter 29.

Four columns, and only the last one decides:

The rule that falls out of it, and it is short enough to keep: start with the prompt; add retrieval when the facts move; fine-tune only when you have measured that the thing you still lack is a shape, not a fact — and price the answer, not the prompt, before you do.

The uncomfortable version, for anyone who arrived having already decided: on the measured case in this chapter fine-tuning is the cheapest route above four thousand questions a month, and on the facts it still cannot beat answering CLAUDE.md to everything.

Every price here has been per token, and every route a different way of arranging tokens. That is about to stop being true.

Chapter 21 leaves text. An image entering a model is not a string but a grid of patches with a token count you did not choose; a spoken minute is billed by the second at one provider and by the audio token at another; synthetic speech is sold by the character, transcription by the minute, raw compute by the GPU-second. The question this chapter answered with one cost function — which is cheaper? — cannot even be asked until the units match, and no calculator on the internet normalises them.

It is also where training turns up again: an image adapter with a trigger word, and a voice cloned from a sample. Which raises the question the next chapter opens with, and it is not rhetorical: if fine-tuning a language model is almost always the wrong purchase, why is fine-tuning an image model almost always the right one?

Every price, threshold and multiplier in this chapter was read from the provider's own page on 7 September 2026 and is quoted with that date, because all of them will move. The measured figures — token counts, chunk sizes, retrieval sizes, training loss, scores, latencies and version-history counts — were produced on one machine on the same day and are reproducible from the corpus described above.

The local experiments used Qwen/Qwen2.5-0.5B-Instruct with greedy decoding, so they reproduce exactly; the adapter is the twelve-line class printed above, at rank 8 over q_proj and v_proj. The corpus is the tracked Markdown documentation of one working software repository, excluding two append-only logs, and its change rate was counted from that repository's version history.