Chain of Thought, RLVR and Test-Time Compute, Measured

The temptation is to say the second version "thought about it". Resist that, because the mechanism is both simpler and more useful to know.

A transformer does a fixed amount of computation per token generated. One forward pass: the same layers, the same matrices, the same number of operations regardless of whether the question is what is 2+2 or prove this theorem. There is no dial inside the model for "try harder on this one".

So when a model is asked for an answer immediately, the entire computation available to it is one forward pass. Every intermediate quantity has to fit in the activations of that single pass, and anything it cannot compute there, it cannot compute.

Emitting tokens changes that, and it changes it in two distinct ways worth separating:

• More computation. Each generated token is another full forward pass. A hundred and forty-five tokens of working is a hundred and forty-five times the arithmetic of answering straight away.
• Externalised memory. The tokens are written into the context, so the next pass can read them. 5 × 13 = 65 becomes a fact in the input, not a value the model must hold in an activation and carry forward. The model is using its own output as a scratchpad.

That second point is the one people miss, and it explains why the working has to be written down to help. A model asked to "think about it silently and then answer" has nowhere to put the thought.

None of this requires anything mystical, and it makes a firm prediction: chain of thought should help most on problems with serial structure — where step two needs the result of step one — and least on problems that are a single lookup. That is exactly what the literature finds, and it is why "think step by step" does nothing for what is the capital of France.

The technique arrived in 2022 in two pieces. Wei et al. showed that including worked examples in the prompt — demonstrations where the answer is preceded by reasoning — produced large gains on arithmetic and commonsense benchmarks.¹ Kojima et al. then showed something stranger: you do not need the examples. Appending "Let's think step by step" to a zero-shot prompt captures much of the same gain.²

The second result is the one that tells you what is going on. If a magic phrase unlocks the behaviour, the behaviour was already in the model — pretraining is full of worked solutions, and the phrase is a pointer to that region of the distribution. Chain of thought did not teach the model anything. It selected something the model already had.

That framing also predicts the technique's eventual obsolescence, which we come back to at the end of the chapter.

The obvious next move: if one chain of reasoning can be wrong, sample several and take the majority answer. That is self-consistency.³ It is a strictly larger spend — $n$ full generations instead of one — and the intuition is that wrong answers scatter while right ones agree.

Measured on 16 of the same problems, sampling at temperature 0.8, majority vote over $n$ chains:

Sixteen problems is a small denominator, and Chapter 4's rule applies to this table as much as to any other. 13 of 16 is 81 % with a 95 % Wilson interval of [57, 93]; 16 of 16 is 100 % with [81, 100]. Those overlap. Read the shape of the curve, which is the finding; do not read the exact rung where it flattens, which sixteen problems cannot locate.

Two things in that table, and the second one is not what I expected.

The curve flattens at $n = 3$. By the third sample the accuracy is at its ceiling and the remaining two samples buy nothing while costing 172 tokens each, 345 between them. That is the shape of every self-consistency curve reported in the literature, and it is much earlier than the "more samples is more better" framing suggests.

And greedy decoding was already at 100 %. Look back at the top of the chapter: one chain, no sampling, 145 tokens, 24/24. Sampling at temperature 0.8 dropped accuracy to 81 %, and self-consistency needed three generations to climb back to where a single greedy pass already was — at 3.6 times the tokens, or six times if you run the sweep to five without knowing where it flattens.

That is not an argument against self-consistency. It is a precise statement of what it does: temperature buys diversity by injecting errors, and voting removes the errors it just injected. On problems where greedy decoding fails — where the single most likely chain leads somewhere wrong and a less likely one is right — that trade pays, and it is why the technique exists. On problems where greedy already succeeds, it is a way to spend six times the budget to break even.

Nobody publishes the second case, which is why it is worth measuring on your own task before adopting the technique. These are easy two-step problems for a small model; that is the regime where the answer comes out this way.

Everything so far happens at prompt time on a model that was never specifically trained for it. The shift that produced the current generation of reasoning models was to move it into training — and the key that made that possible is narrower than it sounds.

Chapter 11's post-training needed human preferences, because "was this a good answer?" has no programmatic answer. But for some questions it does. A mathematical answer either equals the correct value or it does not. Code either passes the tests or it does not. A proof either checks or it does not.

For those domains you can replace the reward model with a verifier, and everything downstream gets better at once: no annotators, no Bradley–Terry fitting, no reward hacking of the kind measured in Chapter 11 — because you cannot flatter a unit test. This is reinforcement learning from verifiable rewards, and it is the setting GRPO was built for: sample a group of solution attempts to the same problem, check each one, and use the group's mean score as the baseline. No critic, no annotator, no reward model. Just a program that says right or wrong.

The result that reframed the field came from DeepSeek in early 2025.⁶ They took a base model and applied reinforcement learning with verifiable rewards directly, with no supervised fine-tuning stage first — the stage Chapter 11 presents as the foundation of everything. Long chains of reasoning emerged anyway. So did behaviours nobody trained for: the model began re-checking its own steps and, in the paper's most-quoted passage, spontaneously reconsidering an approach mid-solution.

The honest reading is not that reasoning is magic. It is that when the only thing rewarded is being right, and being right on a hard problem requires working through it, then working through it is what the optimiser finds — including the parts of working through it that humans also do, because they are what the problem requires rather than what anyone taught.

The practical consequence of all of this is that a reasoning model produces tokens you asked for and tokens you did not, and you pay for both.

Providers handle this differently, and the difference matters:

• Most APIs count reasoning tokens inside the output token count. Your bill and your max_tokens limit both include the thinking you never see.
• Google's Gemini reports thinking tokens as a separate field, outside the standard output count.

That is a genuine incompatibility between two ways of counting the same thing, and any code that computes cost or enforces a budget across providers has to normalise it. Chapter 16 is where that becomes money, and Chapter 23 where it becomes a budget you can enforce.

The other consequence is a latency one that surprises people the first time. A reasoning model's time to first visible token includes all of its thinking, so a request that streams nothing for eight seconds and then answers in one is not a hung connection — it is the model working. Any interface that shows a spinner with no explanation for eight seconds has a design problem, not a networking one.

A closing warning, because it is the most common way this chapter's material is applied wrongly.

Everything in the first half is a technique for making a model that was not trained to reason produce reasoning anyway. Models trained with RLVR already do it: they emit their own working, at their own length, before answering. Telling such a model to think step by step is at best redundant and at worst harmful — it can produce a short, prompt-shaped chain in place of the longer one the model would have generated on its own, and some providers document exactly this.

The same applies to elaborate reasoning scaffolds built in application code. A prompt that walks a model through a decision tree it already navigates internally is spending your tokens to constrain a behaviour that was trained in. This is the first appearance of a theme that runs through the rest of the course: techniques that were essential in 2022 became superstition by 2025, and the only way to tell which is which for your model, today, is to measure both.

Chapter 15 is where that measurement becomes a discipline rather than an opinion.

Reasoning has an uncomfortable property: it is the one capability whose cost scales with how hard the question is. A model that thinks for nine hundred tokens does nine hundred forward passes, keeps a growing cache in memory for all of them, and holds a GPU for the duration.

That makes the economics of serving a reasoning model sharply worse than serving a chat model, and it turns a set of implementation details into the difference between a viable product and an unviable one: how the cache of past keys and values is stored and reused, how many requests can share a forward pass, and how much precision the weights actually need.

Chapter 13 is the last one where the model is an object in your memory rather than a service behind a port, and it is about making that object cheap enough to serve. It also cashes a promise from this chapter: speculative decoding, which produces several tokens for roughly the price of one by having a small model guess and a large one check — a trick that only makes sense once you have seen how much of a forward pass is spent waiting for memory rather than doing arithmetic.

All measurements in this chapter come from Qwen/Qwen2.5-0.5B-Instruct on 24 generated two-step word problems, greedy decoding except where sampling is stated, with zero truncated generations at the token caps used. They are reproducible, and they are a small model on easy problems: read the self-consistency result as a demonstration of the mechanism, not as a benchmark. Chapter 18 of the CS229 lecture notes and chapter 12 of the Hugging Face LLM Course both cover this material with larger models and proper benchmarks.