Advantage: how AI training learns from better-than-expected answers

What the advantage is

When a chatbot is trained with rewards, every answer it writes gets a score. The advantage is how much better, or worse, an answer scored than was expected for that same question. It is not the score itself. It is the surprise: above expectations or below them.

What the video shows

The video looks at a situation where every answer scores about eight. With scores that similar, the raw score on its own teaches the model nothing. So training looks at how much better than expected each answer did, and makes the answers that beat expectations more likely to be repeated.

Why raw scores fail

Imagine the model writes four answers to one question, and they score 8.1, 8.4, 8.0 and 8.3 (made-up numbers, for illustration). If you trained directly on those numbers, you would push all four answers up by almost the same amount, because all of them are "about 8". That shared 8 swamps everything. The small differences, which are the only part that says which answer was actually better, get drowned out.

Now subtract what you expected. The simplest estimate is the average, 8.2. The four answers become −0.1, +0.2, −0.2 and +0.1. Suddenly the message is clear: the second and fourth answers beat expectations and should become more likely; the first and third fell short and should become less likely. The expected score is called the baseline, and what is left after subtracting it is the advantage.

An everyday example

Think of two exam questions. On an easy one, where almost everyone gets 9 out of 10, scoring 8 is a slightly disappointing result. On a hard one, where the usual score is 3, scoring 6 is excellent. The raw marks say 8 beats 6. Measured against what was expected for each question, the 6 is the real achievement. The advantage works the same way: it always compares an answer with what was expected for that particular question.

Where the baseline comes from

There are two common ways to get the expected score:

• A critic. In the method called PPO, a second model is trained just to predict how well the chatbot usually does on a given question.
• A group average. A newer method called GRPO drops the critic. It asks for several answers to the same question and uses their average as the baseline, much like the four-answer example above.

Either way, the training step, known as the policy gradient (the "policy" is simply the chatbot), raises the odds of answers that scored above the baseline and lowers the odds of those below it. The bigger the advantage, the bigger the push.

Why it matters

When every answer gets a similar score, a signal buried under a big shared number is almost no signal at all. The advantage is the small step that turns "every answer is about 8" into "this one did better than usual, do more of that". Both PPO and GRPO, two of the main ways chatbots are trained with rewards, depend on it.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.