PPO: the four-model recipe that taught chatbots from preferences

What PPO is

After a chatbot has learned to follow instructions from written examples, a second stage teaches it from human preferences: people compare two answers and say which one is better. PPO, short for Proximal Policy Optimization, is the algorithm that made that stage work at scale. It produced InstructGPT and the assistants that descended from it.

The name gives away the main idea. "Proximal" means nearby: the model is improved in small steps that never stray far from where it currently is.

What the video shows

The video presents PPO as the classic, heavyweight recipe for training chatbots on human preferences. It keeps four AI models running at the same time and moves in small, careful steps. It works, but it is heavy and fragile.

The four models, in plain words

• The policy: the chatbot being trained. "Policy" is the training word for whatever decides what to do next, here which words to write.
• The reference: a frozen copy of the chatbot from before this stage, so the new version can be held close to where it started.
• The reward model: a judge, trained on people's choices, that gives each answer a score.
• The critic: a helper that predicts how well an answer to this prompt would normally score, so the policy can tell whether a particular answer beat expectations.

An everyday example

Imagine a cooking school with a crowded kitchen. The student cooks. A judge tastes each dish and gives it a mark. A tutor predicts what a dish from this recipe usually scores, so everyone knows whether today's plate was better or worse than normal. And the student's original recipe is pinned to the wall, so nobody drifts too far from it. On top of that, the student may only change the recipe a little after each tasting, however excited the judge gets. It works. It is also slow, expensive and easy to knock off balance.

How it works

Each round, PPO does three things:

1. Generate fresh answers. The chatbot writes new answers in the middle of training, and the judge scores them.
2. Compare with expectations. For each answer it works out the advantage: how much better or worse it did than the critic predicted.
3. Take a small, clipped step. Answers that beat expectations become more likely and the rest less likely. Every update has a size limit, so one oversized estimate cannot wreck the model in a single move.

Why it is hard

All four models have to sit in memory at once, and the chatbot has to keep producing new answers during training, which is slow. PPO also has a reputation for instability, and the course says that reputation is earned. It is candid about this: it explains PPO but does not try to build it, because squeezing a faithful version into one chapter would be dishonest.

Why it matters

PPO shows where the cost of preference training comes from: every extra model has to sit in memory, and every fresh answer has to be generated. That makes the lighter methods easy to understand. DPO and GRPO, the next ideas in this series, each remove part of this machinery, and DPO in particular replaced PPO for most purposes.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.