RAG in Production: Chunking, Retrieval and Honest Citations

Everything below is measured against one corpus: the first thirteen chapters of this course — 13 documents, 359,067 characters, 127 sections, with the front matter and bibliographies stripped. It is a real technical corpus, with prose, tables, formulas and code blocks in it, and it is exactly the sort of thing people load into a knowledge base and then complain about.

The ground truth is 32 questions, each paired with a needle: a short verbatim sentence from the corpus that answers it. Each needle appears exactly once in the 359,067 characters, and none is a section heading — that check matters, because a chunker that copies headings into every chunk would otherwise score itself. Each question is asked twice, once in course English and once the way a support ticket phrases it: 64 queries over 32 ground truths.

A retrieval is correct when a returned chunk contains the needle whole. That is the only definition matching what the generator needs: half a sentence in the prompt is not an answer, it is a hazard.

The embedding model is all-MiniLM-L6-v2 — 384 dimensions, mean-pooled and normalised, the contrastively trained model Chapter 8 measured. Indexing the corpus takes 20.8 seconds on a CPU, 22 ms per chunk; embedding one query takes 13 ms.

Six strategies from three independent ingredients. Blind cuts every 512 characters without looking at the text. Boundaries never cuts inside a paragraph, falling back to a sentence boundary only when one paragraph is over budget. Header prefixes each chunk with its document title and section path. Overlap copies the last 64 characters of the previous chunk into the next.

With 64 queries the 95 % Wilson interval on R@20 is [0.471, 0.705] for A and [0.718, 0.901] for E — those do not overlap, but most of the other columns do, and an unpaired table cannot separate them. Every strategy answers the same queries, so the honest test is paired: count each strategy's wins and losses against another and run a sign test on the discordant pairs. Three results survive it.

Blind chunking destroys four of the thirty-two answers outright. Not ranks them badly — destroys them. The needle straddles a 512-character boundary, so no chunk in the index contains it, and the recall ceiling for those queries is zero. No reranker recovers them, no threshold helps, no larger model helps. You cannot retrieve text that is not in one piece anywhere in your index. This is the single most under-reported failure in RAG, because it looks exactly like a bad retriever.

Overlap fixes that and nothing else. Every strategy with overlap loses zero answers, which is what overlap is for. It does not improve ranking: B against A at R@8 is +8/−6, p = 0.79; at R@20 it is +9/−7, p = 0.80. Worse, adding overlap on top of the header actively hurts — F against E is +2/−6 at R@20 — and the reason is mechanical. A chunk's vector is a mean over its tokens, so 64 characters of the previous chunk drag that mean towards the neighbour's topic. Overlap is insurance against a split answer, paid for in precision.

The contextual header is what buys retrieval. E against A is +18/−3 at R@20, p = 0.0015. And the ablation says the boundaries are not doing it: E against C — same cuts, header the only difference — is +12/−2, p = 0.0129. Prefixing "Classification, Cross-Entropy, and How Not to Fool Yourself > How many test examples do I need?" to a paragraph tells the embedding model what the paragraph is about, which the paragraph itself often does not say. It is a pronoun resolver for documents.

Which gives the chunker its shape, and one rule that is easy to get wrong:

export interface Chunked {
  /** What gets EMBEDDED: contextual header + this chunk's own content. */
  text: string;              
  /** ONLY this chunk's own content: what is quoted back to the user. */
  content: string;           
  section: string;
  /** Character range in the document's canonical text. Sliceable. */
  from: number;
  to: number;
}

export function chunkDocument(doc: string, docTitle: string, target = 512): Chunked[] {
  const out: Chunked[] = [];
  const heads = [...doc.matchAll(/^## (.+)$/gm)].map((m) => ({ at: m.index!, title: m[1].trim() }));
  const spans = heads.length
    ? heads.map((h, i) => ({ ...h, end: i + 1 < heads.length ? heads[i + 1].at : doc.length }))
    : [{ at: 0, title: "", end: doc.length }];

  for (const s of spans) {
    const header = s.title ? `${docTitle} > ${s.title}` : docTitle;     
    const skip = /^## .+\n/.exec(doc.slice(s.at, s.end))?.[0].length ?? 0;
    const body = doc.slice(s.at + skip, s.end);
    const origin = s.at + skip;

    // The offset is FOUND in the document, never accumulated: adding up
    // lengths drifts by a character wherever a separator was normalised,
    // and a citation anchor off by one points at the wrong line.
    const emit = (from: number, to: number) => {
      const raw = body.slice(from, to);
      const lead = raw.length - raw.trimStart().length;
      const content = raw.trim();
      if (!content) return;
      out.push({ text: `[${header}]\n${content}`, content, section: s.title,
                 from: origin + from + lead, to: origin + from + lead + content.length });
    };

    let open: [number, number] | null = null;
    for (const m of body.matchAll(/[^\n]([^\n]|\n(?!\n))*/g)) {          // paragraphs
      const [pf, pt] = [m.index!, m.index! + m[0].length];
      if (pt - pf > target) {                                            // one huge paragraph
        if (open) { emit(open[0], open[1]); open = null; }
        let cur: [number, number] | null = null;
        for (const sm of body.slice(pf, pt).matchAll(/[^.!?]*[.!?]*\s*/g)) {
          if (!sm[0]) continue;
          const [sf, st] = [pf + sm.index!, pf + sm.index! + sm[0].length];
          if (cur && st - cur[0] > target) { emit(cur[0], cur[1]); cur = null; }
          cur = cur ? [cur[0], st] : [sf, st];
        }
        if (cur) emit(cur[0], cur[1]);
        continue;
      }
      if (open && pt - open[0] > target) { emit(open[0], open[1]); open = null; }
      open = open ? [open[0], pt] : [pf, pt];
    }
    if (open) emit(open[0], open[1]);
  }
  return out;
}

Two texts, not one. text is what gets embedded, header and all. content is only this chunk's own words, and it is what gets quoted back to the user. Quote text and the citation shows a header that is not in the document at that point — and, with overlap, a repeated tail belonging to the previous fragment. It then displays text that is not where it says it is, which is worse than showing nothing.

The header is not free. Across the 940 chunks it costs 24,213 of the index's 114,275 embedded tokens: 21.2 % of what you pay to embed is a header you wrote yourself. It also pushes chunks against the encoder's window. all-MiniLM-L6-v2 accepts 256 word-pieces; strategy E has 17 chunks over that line and F has 28, every one silently truncated with no warning from anything. Your effective chunk size is not the number in your config — it is the smaller of that and your encoder's window.

Dense retrieval has one systematic weakness and it is not subtle: it matches meaning, so it is indifferent to exactly which string you typed. A part number, an error code, an acronym, a surname — none has a useful meaning to embed, and the nearest neighbour of an error code is every other error code in your corpus.

The classical answer is older than all of this and takes twenty lines. BM25 scores a document by how often the query's terms appear in it, damping each term as its frequency rises and penalising long documents that accumulate matches by sheer length.¹ Term $t$ contributes

where $f_{t,d}$ is the term's count in the document, $|d|$ its length, $\overline{|d|}$ the average length, and $k_1 = 1.2$ and $b = 0.75$ the two conventional constants — $k_1$ setting how fast repetition stops helping, $b$ how hard length is punished.

const toks = (s: string) => s.toLowerCase().match(/[a-z0-9]+/g) ?? [];

export class BM25 {
  private tf: Map<string, number>[] = [];
  private len: number[] = [];
  private idf = new Map<string, number>();
  private avg = 0;
  private k1: number; private b: number;
  constructor(docs: string[], k1 = 1.2, b = 0.75) {
    this.k1 = k1; this.b = b;
    const df = new Map<string, number>();
    for (const d of docs) {
      const t = new Map<string, number>(); const ws = toks(d);
      for (const w of ws) t.set(w, (t.get(w) ?? 0) + 1);
      for (const w of t.keys()) df.set(w, (df.get(w) ?? 0) + 1);
      this.tf.push(t); this.len.push(ws.length);
    }
    this.avg = this.len.reduce((a, b) => a + b, 0) / this.len.length;
    const N = docs.length;
    for (const [w, n] of df) this.idf.set(w, Math.log(1 + (N - n + 0.5) / (n + 0.5)));
  }
  scores(query: string): number[] {
    const q = toks(query);
    return this.tf.map((tf, i) => {
      const L = this.len[i]; let s = 0;
      for (const w of q) {
        const f = tf.get(w); if (!f) continue;
        s += (this.idf.get(w) ?? 0) * (f * (this.k1 + 1)) /
             (f + this.k1 * (1 - this.b + (this.b * L) / this.avg));
      }
      return s;
    });
  }
}

Over 940 chunks that scores a query in 1.14 ms with no index at all beyond two hash maps. And it is not a museum piece:

BM25 more than doubles the dense retriever's top-1 accuracy on this corpus, and loses to it badly by rank 8. They fail on different queries, which is the entire argument for running both.

Fusing them is the one place where the obvious approach is wrong. Cosine distances and BM25 scores are not on the same scale, are not bounded the same way, and normalising them per query makes the weight depend on how good the best hit happened to be. Reciprocal rank fusion throws the scores away and keeps only the ranks:²

/** Reciprocal rank fusion: ranks, not scores. Nothing to calibrate. */
export function rrf(lists: number[][], k = 60): number[] {
  const acc = new Map<number, number>();
  for (const list of lists)
    list.forEach((id, r) => acc.set(id, (acc.get(id) ?? 0) + 1 / (k + r + 1)));
  return [...acc.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

And here the honest reading of the table matters more than the table. Hybrid beats BM25 at R@4 by +10/−3, p = 0.09. It beats dense by +10/−6, p = 0.45. On this corpus, with 64 queries, hybrid retrieval is not distinguishable from dense retrieval. It is better on both point estimates and every recall column, and the evidence does not reach significance. Almost every hybrid-search blog post on the internet reports a table like the one above and no interval; this is what the interval says.

Everything so far is a bi-encoder: the query goes through the model alone, each chunk went through it alone months ago, and the two never meet except as a dot product. That is what makes an index possible — embed once, reuse forever — and it is also the ceiling. The model never looks at the query and the chunk together.

A cross-encoder does exactly that: it takes the pair as one input and returns a relevance score. Nothing can be precomputed, so it cannot rank an index — but it can rerank a shortlist. Reranking the hybrid top 25 with ms-marco-MiniLM-L-6-v2 moves R@1 from 0.094 (dense) to 0.312 and MRR from 0.280 to 0.447: the largest single improvement in this chapter, and the only one that touches the top of the list rather than the tail.

It costs 569 ms per query on a CPU, against 1.14 ms for BM25 and 0.3 ms for the vector scan. Roughly two thousand times the retrieval cost, for twenty-five documents. That is the whole bi-encoder/cross-encoder trade in one number, and it is why the architecture is always the same shape: a cheap retriever with wide recall, then an expensive scorer on a shortlist you can afford. ColBERT sits between the two, precomputing per-token vectors and doing a late interaction that is cheaper than a cross-encoder and sharper than a dot product.³

Vector databases report distances, and which distance is a configuration option. On normalised vectors the choice is cosmetic, and the identity is worth doing once because everything after it depends on the vectors really being unit. For $\lVert a \rVert = \lVert b \rVert = 1$:

so the cosine distance $1 - \cos\theta$ is exactly $d^2/2$. That is Chapter 1's dot product and norm, cashed. Checked on two real chunk vectors from the index above, and then over 40,000 pairs:

||a|| = 1.000000   ||b|| = 1.000000
L2 = 0.795183   L2^2/2 = 0.316158   1 - cos = 0.316158   diff = 7.66e-08
max |L2^2/2 - (1 - cos)| over 200 x 200 pairs = 8.3e-07

Exact to floating-point noise — and only because the vectors are normalised. Skip the normalisation and the identity is false, your threshold means nothing, and the distance a document reports depends on how long its text was.

Now the number nobody derives. A retriever always returns something: it sorts the whole index and hands you the top of the list, whether or not the answer is anywhere in the corpus. The threshold is the only part of the system that can say no — and to set one you need queries that should get nothing back. Here are thirty: twenty-one about things this corpus genuinely does not cover — streaming, rate limits, prompt caching, JSON schemas, agent loops, vector databases, prompt injection, image generation — and nine about paella, passports and refund policies. Against the same index:

The distributions separate, and they overlap. The worst in-domain query is further from its answer (0.721) than the best out-of-domain query is from an irrelevant paragraph (0.497), so no threshold gets both right. Sweeping it over the real gate — keep at most four chunks, and only the ones under the cut:

Read the last column as bluffs. With no threshold the assistant produces a confident, well-cited answer to "how do I renew my Spanish passport" from a corpus about backpropagation, thirty times out of thirty. At 0.675 it does it ten times out of thirty. At 0.525 it does it twice, and gives up on twelve questions it could have answered.

That trade is a product decision, and the right end of it depends on what a wrong answer costs you. What is not negotiable is the last column existing at all. If you have never measured your retriever against questions it should refuse, you do not have a threshold — you have a number.

Two of the ten bluffs at 0.675 show the two ways this fails.

query: "how much does prompt caching save on a long conversation"
  [1] d=0.497  13-inference-optimization > Prefill and decode are two different machines
  [2] d=0.532  13-inference-optimization > The cache is also the bill

query: "what is the capital of france"
  [1] d=0.671  12-reasoning > The model does not think. It computes for longer.
      "…it is why 'think step by step' does nothing for what is the capital of France."

The first is a near miss: the corpus explains the KV cache in detail, the query is about the prompt cache, the words are the same words, and 0.497 is closer than most correct in-domain retrievals in the whole experiment. An embedding does not know that two caches with the same name are different machines. The second is a literal match with no answer: the corpus contains the exact phrase "what is the capital of France", used as an example of a question that needs no reasoning. The retriever is right; the answer is not there. Any system that reads "I found something similar" as "I found the answer" will assert Paris on that evidence — or, worse, will not.

A model has no separate faculty for facts. Producing a true sentence and producing a plausible one are the same operation — Chapter 8's next-token prediction — and nothing in that operation marks which is which. The 2025 analysis that reframed this argues that the training and evaluation pipeline actively rewards guessing: benchmarks score with binary accuracy and give no credit for abstention, so a model that answers always outscores an identical model that says "I don't know" when it doesn't, and post-training optimises accordingly.⁶ Hallucination on that reading is not a mysterious defect. It is what you get when you grade a multiple-choice exam with no penalty for a wrong answer.

Watch the shape of it. Asked for eight papers on contrastive sentence embeddings, with identifiers, Qwen2.5-0.5B-Instruct produced eight lines in perfect format. All eight identifiers are well-formed. All eight resolve to real papers on arXiv. Zero of the eight are the paper claimed.

claimed  arXiv:1907.06432 - Contrastive Sentence Embeddings for Text Retrieval
actual   A Neural Turing~Machine for Conditional Transition Graph Modeling

claimed  arXiv:1809.08669 - Contrastive Learning of Sentence Representations…
actual   Collapsing Superstring Conjecture

claimed  arXiv:1807.08669 - Contrastive Learning of Sentence Representations…
actual   Automatic Speech Recognition for Humanitarian Applications in Somali

This is a small model and the rate is its own; a frontier model invents far fewer. The mechanism generalises, and it is the reason for the rule that follows. A validator checking "does this identifier exist" passes all eight, and a user who clicks one lands on a real page from a real archive with no way to tell the mapping was invented. The failure is not in the identifier or the format. It is in the association — precisely the thing a language model produces by plausibility.

So: the model writes [1] and [2], and never writes the link. The numbers refer to fragments the server retrieved, and the server — which knows exactly which document and which offsets each number came from — attaches the document, the label and the URL afterwards. There is nothing for the model to invent because it is never asked for the one thing it would invent.

export function buildContext(question: string, hits: Scored[]) {
  const citations: Citation[] = hits.map((h, i) => ({
    index: i + 1,
    documentId: h.chunk.documentId,
    documentName: h.chunk.documentName,
    locatorLabel: label(h.chunk),
    fragment: `#char=${h.chunk.locator.flow.from},${h.chunk.locator.flow.to}`,  
    quote: h.chunk.content,          // the OWN content, never `text`
    cosineDistance: h.cosineDistance,
  }));
  const blocks = citations
    .map((c) => `[${c.index}] ${c.documentName} - ${c.locatorLabel}\n${c.quote}`)
    .join("\n\n");
  const prompt =
    `Answer using ONLY the numbered sources below. Cite every claim as [n].\n` +
    `If the sources do not contain the answer, say so and stop.\n\n` +
    `SOURCES\n${blocks}\n\nQUESTION\n${question}`;
  return { prompt, citations };
}

Run it on the opening question and the four chunks become a 591-token prompt and a table the model never sees:

[1] 04-classification  How many test examples do I need?      #char=28215,28701  d=0.594
[2] 04-classification  Three splits, and the leak…            #char=20329,20839  d=0.598
[3] 04-classification  How many test examples do I need?      #char=25873,26272  d=0.600
[4] 04-classification  How many test examples do I need?      #char=25554,25871  d=0.605

The locator is the part people skip and then cannot add later. #char=25873,26272 is a range in the document's canonical text; for a PDF the equivalent is #page=12, for audio or video #t=132.4,158.9, for a spreadsheet a sheet and an A1 range. Those two are not inventions — #page= is PDF Open Parameters and #t= is W3C Media Fragments, honoured natively by browsers on video and audio elements. A citation without a locator is a document name, and a document name is not a citation; it is a suggestion that the user go and look.

And when nothing passes the threshold, the pipeline never reaches the model at all:

NO ANSWER: nothing under cosine distance 0.675 for "what is the offside rule in football"
NO ANSWER: nothing under cosine distance 0.675 for "how do i renew my spanish passport"
NO ANSWER: nothing under cosine distance 0.675 for "how do i build an agent loop with tools"

That is a cheaper and more reliable refusal than any instruction in a system prompt, because it is a comparison between two numbers rather than a request to a probabilistic system.

Every measurement in this chapter scores the retriever and never once asks a model to write an answer. That is deliberate, and it is the piece most teams skip.

A RAG system has two failure modes that look identical from outside. The retriever did not find the passage; or it found it and the generator ignored it, contradicted it, or blended it with something it already believed. Score only the final answer and the two are indistinguishable, so you tune prompts against a problem that lives in your chunker. Recall@k, MRR and the answer-destroyed count need no generation call at all, they are cheap enough to run on every deploy, and they are the harness from Chapter 15 with a different scoring function — the same request, deadline, concurrency and tally, over a fixed question set instead of a live conversation.

Report them with intervals. Chapter 4's arithmetic applies unchanged: at 64 queries a recall of 0.5 carries a 95 % Wilson interval of roughly ±0.12, so a strategy four points ahead of another has told you nothing. Use the paired test whenever both strategies answer the same questions, which they always do here — it is what turned "E looks better than A" into p = 0.0015.

And the last honesty: RAG reduces hallucination and does not remove it. Putting the right passage in the prompt does not oblige the model to use it, and the literature has said so since the original paper.⁷ Two things make it worse in production. Long contexts degrade — a model finds information at the start and the end of a long prompt more reliably than in the middle, so twenty chunks instead of four can lower accuracy while raising the bill, an effect measured in Chapter 24. And retrieval can be right and still insufficient, as the two caches above showed. SelfCheckGPT flags claims that do not survive resampling;⁸ Self-RAG trains the model to emit its own retrieve-and-critique tokens;⁹ TruthfulQA made the failure mode legible in the first place.¹⁰ None closes the gap, and a system that presents retrieved text as proof has confused sourced with true.

A retriever is the visible part of a pipeline whose failures all happen earlier, in the dark. Three of them recur.

Extraction is where the content dies. A PDF is not text; it is drawing instructions. Two-column layouts interleave, tables become word soup, page headers repeat into every chunk, and a scanned page has no text at all until OCR gives it some, with a confidence. Everything measured above assumed the extractor did its job; in production it often does not, and the symptom appears as bad retrieval three layers away.

The index is stamped with the model that built it. Embeddings from two models are not comparable — not "less accurate", not comparable, because they are points in different spaces. Change the embedding model and every vector in the store is garbage until it is rebuilt. So the model name, the dimension count, the pipeline version and the extractor version are written beside each document at index time. Without them, on upgrade day, you cannot tell which documents are stale and which are current, and a half-migrated index returns confident nonsense with no error anywhere.

One broken document must not break the folder, and the counters must count what happened. A document that fails extraction ends in a failed state with its reason, visible and retryable, while the other ninety-nine stay searchable; and the number of chunks indexed is written by the server when it finishes, not declared by the client when it uploads. A folder that reports 400 fragments and holds 40 is a lie that surfaces only as an unanswerable question.

The system in this chapter answers questions whose answers are written down. It retrieves them, ranks them, refuses when it cannot, and cites where it looked. That is most of what people want from an assistant over their own documents, and it is bounded in one specific way: retrieval can only return what somebody wrote.

Which leaves the other half. Some of what you want a model to do is not a fact in a document at all — a format it has to hold, a tone, a taxonomy with four hundred labels, a way of deciding that lives in ten thousand past examples and in no paragraph anywhere. Retrieval cannot deliver those, because there is nothing to retrieve; a longer prompt only pays Chapter 16's bill for a description of a skill instead of the skill.

Chapter 20 is that decision — fine-tune, retrieve or prompt — and its finding is that the decision is economic before it is technical: the three are priced end to end on the same question, and the crossover is a token count. The question that opens it is the one this chapter cannot answer. Not where is the answer written, but what do you do when it never was.

Everything measured in this chapter used one corpus and one instrument, and both are reproducible. The corpus is chapters 1 to 13 of this course as they stood on 7 September 2026 — 13 documents, 359,067 characters, 127 sections, front matter and bibliographies removed. Those chapters keep being edited, so applying the same rule today counts a few thousand characters more: the section count is unchanged and so is every conclusion below, but the character total is a snapshot and is labelled as one. The ground truth is 32 questions, each paired with a verbatim sentence that occurs exactly once in the corpus and is never a section heading, asked in two phrasings for 64 queries. Retrieval embeddings are sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, mean-pooled, L2-normalised, 256-token window); reranking is cross-encoder/ms-marco-MiniLM-L-6-v2 over the top 25; the generation example is Qwen/Qwen2.5-0.5B-Instruct with greedy decoding. All timings are single-threaded CPU. No paid API was called to produce this chapter, which is also why every latency here is a local one and is labelled as such.

The chunker shown in TypeScript is the chunker that was measured: the Python instrument implementing the same rule and ts/chunk.ts were compared chunk for chunk over the whole corpus and agree on all 940 chunks, texts and offsets alike. Intervals are Wilson at 95 %; paired comparisons are two-sided exact sign tests on the discordant pairs.

All fourteen identifiers cited above were resolved against the arXiv API and checked title by title on 7 September 2026 — which, given the eight that were not, seemed like the least this particular chapter could do.