GRPO: how an AI learns by comparing its own answers

What GRPO is

GRPO, short for Group Relative Policy Optimization, is a way to train a language model with rewards. It was introduced alongside DeepSeekMath. It starts from the classic recipe, PPO, and deletes one of its four models: the critic, whose only job was to predict how well an answer to a given question would normally score. GRPO gets that expectation another way, by letting the model answer the same question several times and taking the average.

What the video shows

The AI answers the same question several times, and each answer is judged against the group's average. No extra predictor is needed, and that is what made training reasoning AIs practical.

An everyday example

Imagine practising darts. You throw eight darts at the same board, from the same spot, in the same light. To decide which throws were good, you do not need a coach telling you what a typical throw ought to score. You just compare each throw with the average of your own eight. The throws above that average show you what to repeat. The ones below it show you what to drop.

That is the whole trick. The yardstick for each answer is simply the other answers to the very same question, from the same model at the same moment.

How it works

For each question in training:

1. Sample a group. The model writes several answers to the same question.
2. Score them. Each answer gets a reward, a number saying how good it was.
3. Compare with the siblings. The group's average reward becomes the baseline, the "normal" score for this question. An answer's advantage is simply how far above or below that average it landed.
4. Update. Answers above the average become more likely; answers below it become less likely.

The idea of scoring an answer against an expectation, the advantage, has its own video in this series. What GRPO changes is only where the expectation comes from.

The trade-off

GRPO swaps a whole model for a bigger batch. You no longer need to train and store a critic, but you do need to produce a whole group of answers for every question. Unlike DPO, which learns only from pairs of answers that already exist, GRPO keeps PPO's habit of generating fresh answers during training.

| | Critic model | Fresh answers during training | |---|---|---| | PPO | Yes | Yes | | GRPO | No | Yes, several per question | | DPO | No | No |

Why it matters

GRPO is what made training on verifiable rewards practical: rewards a computer can check on its own, such as whether the final answer to a maths problem is right. That is the route to the reasoning AIs the video mentions, and the course covers it in depth in its next chapter. Dropping a model you would otherwise have to train and keep in memory turned an expensive idea into a workable one.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.