DPO: training a chatbot on preferences without a judge model

What DPO is

To teach a chatbot what people actually prefer, annotators compare pairs of answers and mark the better one. The classic recipe, PPO, turns those choices into a separate judge model first, and only then trains the chatbot to please that judge. DPO, short for Direct Preference Optimization, skips the judge. It learns directly from the pairs themselves: this answer won, that one lost.

What the video shows

The video shows the method learning straight from pairs of better and worse answers instead of training a separate judge. It needs half the machinery, and it concentrates its effort on its own mistakes.

An everyday example

Imagine learning to write good work emails. One way is to hire a grader, train them on your manager's taste, and then write draft after draft for the grader to mark. The other way is to sit down with a stack of pairs your manager has already sorted, "this one, not that one", and study the differences directly. Like a good flashcard app, you would spend most of your time on the pairs you would have got wrong and only glance at the ones you already understand.

How it works

The shortcut comes from a piece of maths. The goal PPO chases (score well, but do not drift too far from the starting model) turns out to have an exact best solution. Rearrange that solution and the judge's scores can be written using only the chatbot and its starting copy. Put that back into the usual formula for learning from pairs and the judge drops out completely.

What is left is an ordinary training loss on preference pairs. (A loss is the number that training tries to push down.) In plain words, DPO rewards the chatbot for leaning towards the winning answer more than its starting copy did, compared with how much it leans towards the losing one. Because the starting copy is built into the loss, DPO needs no separate fine for drifting away from it.

The practical result:

• No judge and no critic. Two models in memory instead of PPO's four.
• No answer generation during training. It learns from pairs that already exist.

It focuses on its own mistakes

The course measures how hard a single pair pushes on the model, depending on what the model currently believes about it:

| The model currently… | Loss | Push on the model | |---|---|---| | strongly prefers the winner | 0.5130 | 0.0401 | | matches its starting copy | 0.6931 | 0.0500 | | strongly prefers the loser | 1.0055 | 0.0634 |

Pairs the model already gets right barely move it. Pairs it gets backwards drive the update. Nobody has to schedule this; it falls straight out of the formula. It does the work that PPO left to a whole critic model and its estimates of which answers beat expectations.

There is also a handy check. A model identical to its starting copy should sit at a loss of exactly ln 2, about 0.693, which is the loss of a fair coin flip between the two answers. If an implementation does not land there, something is wrong.

Why it matters

DPO replaced PPO for most purposes. It keeps the core benefit, learning from what people prefer, while dropping half the models and the slow step of generating new answers mid-training. GRPO, the next idea in this series, solves the same problem another way.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.