Process reward: grading an AI's working, not just its final answer

What a process reward is

When people train an AI model to solve problems, they give it a reward for good work, and the model slowly shifts towards whatever earns more reward. This is how today's reasoning models, the ones that write out their working before they answer, are sharpened. The big question is what counts as good work.

An outcome reward looks only at the final answer: right or wrong. A process reward looks at every step of the working and scores each one. It is the difference between a teacher who checks the answer box and a teacher who reads the whole page.

What the video shows

The video compares the two ways of grading. Rewarding only the final answer lets lucky wrong reasoning slip through: a solution that lands on the right number by a broken route earns the same reward as a correct one. Grading every step does better on hard maths problems, but it has a price. Someone has to label the steps, and one research dataset contains 800,000 of them, each labelled by a person.

An everyday example

Imagine a student asked "what is 2 × 2?" who adds instead of multiplying. They write 2 + 2 = 4 and get full marks. The answer is right, the method is wrong, and nobody notices. Next week the question is 3 × 3, they write 6, and now the mistake shows.

A teacher who marks the working would have caught it the first time, because the line "2 + 2" is the error, whatever the final number says. That is exactly the gap between the two kinds of reward.

How it works

| | Outcome reward | Process reward | |---|---|---| | What gets checked | Only the final answer | Every step along the way | | Cost | Cheap: compare the answer with the correct one | Expensive: people must label many steps first | | Weak spot | Rewards right answers reached by wrong reasoning | The labelling effort |

With an outcome reward, the check can be as simple as comparing two bits of text. That is why it is so attractive, and it is the kind of automatic right-or-wrong check that verifiable rewards rely on. The hole is that the model is free to learn working that looks convincing and just happens to end in the right place.

A process reward closes that hole by grading each line. To do it automatically, a research team (Lightman and colleagues) had people label 800,000 individual reasoning steps by hand, then used those labels to train a separate model whose job is to score steps. On hard maths, grading the steps clearly beat grading only the answer.

Why it matters

If only the final answer counts, a model can be praised for reasoning that is actually broken, and nothing in its training tells it so. That broken method can then fail on the next problem, just as the student's did. Grading each step closes the loophole: a wrong step costs points even when the final number happens to be right.

The trade-off is easy to state. Outcome rewards are cheap but leaky. Process rewards are more reliable, but somebody has to do an enormous amount of careful labelling first. And the lesson reaches beyond AI training: a correct answer on its own is not proof of sound reasoning.

Learn it step by step in Chapter 12 of our free course AI From Scratch: Chain of Thought, RLVR and Test-Time Compute, Measured.