Reward model: the AI judge that learns our tastes, biases included

What a reward model is

One way to make a chatbot helpful is to have people pick the better of two answers and train the chatbot toward the winners. There is a practical problem: training needs millions of judgements, and no team of people can make that many. So a separate AI is trained to stand in for them. That AI is the reward model. You show it an answer, and it gives back a single number: its estimate of how much people would prefer it. The chatbot is then trained to earn high numbers.

What the video shows

The video is about an AI judge that learns from people choosing between answers, and the problem that comes with it: if the people lean toward longer replies, the judge learns that lean too, even when the extra length adds nothing.

How it learns

The reward model never sees a score written by a person. It only sees pairs: this answer won, that one lost. To learn from them it uses an idea from 1952 called the Bradley–Terry model. It turns the gap between two scores into a probability: the further the chosen answer's score is above the rejected one's, the more certain the judge is that the chosen answer should win. Training nudges the winners' scores up and the losers' down until the judge agrees with the people as often as it can.

Notice what that means. The judge only ever learns differences. A score of 7 means nothing on its own. It only becomes meaningful next to another score: an answer at 7 is expected to beat an answer at 5 more often than not. It is a bit like a chess rating, which is built only from wins and losses: one player's number tells you little until you compare it with an opponent's.

An everyday example

Imagine a trainee teacher learning to mark essays by watching which of two essays the head teacher prefers. The head teacher cares about ideas and accuracy, but when two essays are close, they lean slightly toward the longer one. The trainee cannot tell which part of each choice is quality and which part is habit. So they copy all of it, and "longer is better" quietly becomes one of their marking rules.

The experiment in the course

The course tests exactly this with a simulation. The true quality of an answer depends only on two things, being useful and being correct. Length is worth nothing. The simulated person choosing, the annotator, tends to favour the longer of two answers when they are otherwise about equal, a well-known human habit. After training the judge on 2,000 comparisons, here is how much weight it gives to length:

| Annotator's lean toward length | Weight the judge gives to length | |---|---| | 0.0 (none) | +0.01 | | 0.3 | +0.15 | | 0.6 | +0.27 | | 1.2 | +0.59 |

For comparison, the weights for being useful and being correct stay at about +1.00 in every case. The judge is not broken. It learned what it was shown, faithfully, including the part that has nothing to do with quality.

Why it matters

A reward model does not measure what is good. It measures what the people choosing picked, with their habits built in. That matters because a much bigger model, the chatbot, is about to be trained to push that number as high as it can. Any small bias in the judge becomes something the chatbot can learn to exploit, a problem known as reward hacking.

Learn it step by step, with the full simulation, in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.