RLHF: why people pick the better answer instead of writing it

What RLHF is

RLHF stands for reinforcement learning from human feedback. In plain words: people look at two answers from an AI and say which one is better, and the AI is then trained to produce more answers like the winners. It is the second stage of turning a raw language model into an assistant. It comes after a first stage in which the model learns from example answers that people wrote by hand.

What the video shows

The video is built on a simple observation: writing an ideal answer is slow, but choosing the better of two answers is quick. Those quick choices are used to train an AI judge, and the chatbot then learns to win that judge's approval.

Why comparing beats writing

The first stage has a ceiling, and the ceiling is the data. Someone has to write every ideal reply. For most interesting questions that is hard, slow and expensive, and at the end you have exactly one answer and no reliable way to prove it is a good one.

People are far better at comparing. Put two replies side by side and an annotator (the person doing the judging) can usually say which is better within a few seconds, even without being able to write either reply. The whole second stage is built on that fact.

This is also where many explanations go wrong. In this stage, humans are not writing the chatbot's answers. They are choosing between them.

An everyday example

Think of an eye test at the optician. Nobody asks you to describe your own prescription. Instead you hear, again and again: "Clearer with lens one, or lens two?" You could never write the prescription yourself, but you can answer that question reliably, and enough of those answers add up to the right glasses. RLHF uses the same trick.

How it works

The data is simple: a question, two replies, and which one won. That kind of data cannot be used the way the first stage used its examples, because there is no single correct text for the model to copy. So it takes a different route, in two steps:

1. Train a judge. A separate model, called a reward model, learns from many of these choices to give any reply a score.
2. Let the chatbot practise against the judge. Using reinforcement learning, which means learning by trial and reward, the chatbot writes answers, the judge scores them, and the chatbot is nudged toward the answers that score higher. The classic method for this step is called PPO.

Two words from reinforcement learning make the rest of the topic easier. The policy is simply the chatbot itself: given the conversation so far, it chooses what to write next. The reward is the score for a whole finished answer, not for each word, because you cannot tell whether one word was good until you have read the complete reply.

Why it matters

RLHF is how chat assistants get shaped by what people prefer, not only by examples. The flip side is that they end up tuned to what the human raters preferred, which is not always the same as what is true or useful. Much of the rest of the chapter, from how the judge learns to what happens when a chatbot learns to game it, grows out of exactly this point.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.