From Base Model to Assistant: SFT, RLHF, DPO and GRPO

The first step is the least glamorous and does most of the work. Collect examples of instructions paired with good responses, and continue training on them with exactly the loss from Chapter 8 — predict the next token — but only on the response part. This is supervised fine-tuning, or SFT.

Nothing new is being taught about language. What is being taught is a format: that text of this shape is followed by text of that shape, and then it stops. Look again at the base model's failure. It answered the question in the first sentence and then could not stop, because nothing in its training ever marked the end of a response. Stopping is a learned behaviour.

That is also why the model needs to be told where the boundaries are, which is what a chat template is:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

Those <|im_start|> and <|im_end|> markers are real tokens in the vocabulary, added before fine-tuning, and the model saw millions of them in exactly these positions. They are how it knows whose turn it is and where a turn ends.

Skip the template and hand the model a bare question, and you are giving it a sequence it has not seen in training. Measured, same model, same question, same greedy decoding:

 The capital of France is Paris.

To verify this, I will use a simple
Python code snippet to confirm that
Paris is indeed the capital city of
France.

[...and then it starts writing a
Python script to check its own answer]

The capital of France is Paris.

The answer is right in both, but without the markers the model drifts into writing Python to check itself, because the prompt it received resembles nothing it was fine-tuned on. This is the single most common cause of "the model got dumber when I called it directly": the template is not decoration around the model, it is part of the model, and a wrong template is a silent degradation with no error attached.

SFT has a ceiling, and the ceiling is the data. To fine-tune on a demonstration you need somebody to write the ideal response — and for most interesting questions, writing a good answer is hard, slow, expensive, and produces exactly one answer whose quality you cannot verify.

What people are good at is comparison. Shown two responses, an annotator can reliably say which is better in a few seconds, without being able to produce either. This is the fact the entire second stage is built on, and it is the part most explanations get backwards:

Humans do not write the answers. They rank pairs.

So the data is pairs — a prompt, two responses, and which one won. That cannot be plugged into a next-token loss, because there is no target sequence. It needs a different machine.

You cannot ask a human to score every response during training — that is millions of judgements. So you train a model to imitate the humans: a reward model that takes a response and returns a scalar.

Training it from comparisons uses a result from 1952. The Bradley–Terry model² says that if two items have latent strengths, the probability that one beats the other is the logistic function of their difference. Turn that around and it becomes a loss: given that a human preferred $y_w$ over $y_l$, maximise

which in code is the entire training loop:

loss = -F.logsigmoid(reward(chosen) - reward(rejected)).mean()   

Notice what the model never sees: an absolute score. It only ever learns differences, which is exactly what the data contains.

Now the part worth measuring. A reward model learns what the annotators rewarded, and annotators are people. Here is a simulation where the true quality of a response depends only on being useful and correct — length is worth nothing — but the simulated annotator has a mild preference for longer answers when everything else is close, which is a well-documented human bias. Train the reward model on 2000 comparisons and read its weights:

The reward model is working perfectly. It has faithfully learned the preferences it was shown — including the part of those preferences that has nothing to do with quality. A reward model is not a measure of good; it is a measure of what the annotators picked, and every bias in the annotation pool is now a coefficient in a differentiable function that a much larger model is about to optimise against.

Which brings us to what happens when you optimise it. Give the policy a fixed budget of effort to spend across the response's properties, with a realistic asymmetry: being useful and being correct are expensive, and being longer is cheap — you just keep writing.

Reward per unit of effort, for the model trained above: useful 8.26, correct 8.31, length 31.70. Length pays nearly four times better than correctness, not because the reward model is broken, but because it is cheap.

Optimise against that reward and watch both numbers:

The reward went up by a factor of 2.5. The thing the reward was supposed to measure went to zero. The policy discovered that it could score enormously well by writing at length and saying nothing, and no part of the training loop had any way to notice, because the reward model is the definition of good inside the loop.

This is reward hacking, and if you have ever wondered why chat models are so verbose, this table is a large part of the answer.

The standard defence is to penalise the policy for moving too far from where it started, measuring distance with the KL divergence from Chapter 4:

The reference $\pi_{\text{ref}}$ is the SFT model — the policy before the reinforcement stage. The claim is that this prevents the model from wandering off into degenerate behaviour. Let us find out how much of that claim survives measurement. Same setup, sweeping $\beta$:

Read the last row against the rest. At $\beta = 0$ and $\beta = 1$ the penalty does nothing at all: the reward is worth so much more than the KL that the optimiser pays the fine and hacks anyway. Between 5 and 15 the behaviour swings. And by $\beta = 60$, true quality has climbed back to 1.769 — which is still below the 1.791 the reference model had before any of this started.

One caveat before that number is quoted anywhere: the 1.791 of the last row and the 0.974 that the first table gives the starting policy are two different measurements of the same pre-RL model, taken by the two experiments separately. Compare rows within a table, never across them — the conclusion of each table stands on its own rows, and neither depends on the other's baseline.

So the honest summary is not "the KL penalty prevents reward hacking". It is:

The KL penalty does not prevent reward hacking. It limits how far the policy can move from the reference — and since the failure requires moving, that helps. But it is a leash, not a corrective: at low $\beta$ the leash snaps, and at high $\beta$ you get the reference model back and the whole expensive stage bought nothing.

The useful band is narrow, its location depends on the reward model, and there is no way to find it except by looking. That is why the reference model must be good — the KL is a floor at the reference's quality, not a ceiling on the failure — and it is a large part of why this stage is difficult in practice rather than in principle.

The algorithm that made this work at scale is Proximal Policy Optimization.³ In one paragraph: it estimates the advantage of each response, updates the policy to increase the probability of above-baseline responses, and clips the size of any single update so a large advantage estimate cannot destroy the policy in one step. Applied to language models⁴ it means keeping four models in play at once — the policy, the reference, the reward model, and a critic — with the policy generating fresh samples throughout training.

It works, it produced InstructGPT and everything descended from it, and it is genuinely difficult: four models in memory, sampling in the training loop, and a reputation for instability that is deserved. Pretending you can implement it in a blog post would be dishonest, so this chapter does not.

What replaced it for most purposes came from noticing something. The KL-regularised objective above has a closed-form optimal policy, and that expression can be inverted: the reward can be written in terms of the optimal policy and the reference. Substituting that back into the Bradley–Terry loss makes the reward model disappear entirely. What remains is a supervised loss on preference pairs — no sampling, no critic, no reward model, two models in memory instead of four.

That is Direct Preference Optimization,⁵ and it is two lines:

def dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1):
    """pi_* and ref_* are summed log-probabilities of a full response."""
    logits = beta * ((pi_w - ref_w) - (pi_l - ref_l))   
    return -F.logsigmoid(logits)                        

Read what it says. The quantity being pushed up is how much more the policy prefers the winner than the reference did, minus how much more it prefers the loser. The reference is not a penalty bolted on afterwards — it is inside the loss, which is why DPO does not need a separate KL term.

The property that matters most is in the gradient. Evaluate the loss and its gradient on the same pair in five different states of the policy:

The gradient grows as the policy gets it more wrong. Pairs the model already handles contribute almost nothing; pairs it gets backwards dominate the update. DPO weights every example by how wrong the policy currently is, automatically, with no scheduling — and that self-weighting is the mechanism doing the work that PPO's advantage estimate and critic were doing. (The loss at the third row is exactly $\ln 2$, which is the anchor to check any implementation against: a policy identical to its reference has learned nothing and should sit at $\ln 2$.)

GRPO⁶ takes a different route out of the same problem. It keeps the sampling loop but deletes the critic: instead of training a model to predict the baseline, it samples a group of responses to the same prompt and uses the group's mean reward as the baseline directly. The advantage of a response is how much better it was than its siblings. That trades a whole model for a larger batch, and it is what made verifiable-reward training — the subject of Chapter 12 — practical.

Two things to carry forward.

The first is that this stage has a cost, and it shows up as capability. Models often get measurably worse at some benchmark tasks after alignment training — the alignment tax — because the objective changed: a response that is safe, hedged and formatted well is not always the response that maximises accuracy. Some of that gap has been engineered away, and some of it is a real trade rather than a bug to fix.

The second is the question the word aligned hides. Aligned with whom? The chain is: a company writes guidelines, contractors interpret them, their comparisons train a reward model, the reward model shapes a policy, and the policy answers a question from someone who saw none of it. Every link is a choice made by specific people, and none of the algorithms in this chapter has any opinion about whether those choices are good.

That is not a rhetorical flourish. It is the concrete reason two frontier models refuse different requests, why the same model changes its mind between versions, and why "aligned" is a description of a process rather than a property of an artefact. The mathematics in this chapter is settled. That part is not.

Post-training taught the model to answer. It did not teach it to think before answering, and the two are different in a way that turns out to be trainable.

Chapter 12 is about what happens when you let a model spend more computation on a hard question at answer time rather than at training time — chain of thought, reinforcement learning from verifiable rewards, and the reason a model that shows its working is not merely explaining itself but computing differently. It also cashes the debt from this chapter: GRPO exists in it, doing the job PPO's critic used to do, on rewards that need no annotator at all because a proof either checks or it does not.

The generations above come from gpt2 and Qwen/Qwen2.5-0.5B-Instruct with greedy decoding, so they reproduce exactly. Chapter 11 of the Hugging Face LLM Course walks through SFT and DPO with trl and peft if you want to run the real thing rather than the simulation; chapter 7 of Sebastian Raschka's Build a Large Language Model (From Scratch) implements instruction fine-tuning end to end without a library.