KL penalty: a leash on AI training, not a cure for cheating

What the KL penalty is

When a chatbot is trained to chase a judge's score, it can learn to cheat: to find cheap tricks that raise the score without making its answers any better. That is reward hacking, the previous idea in this series. The KL penalty is the standard defence against it. During training, the chatbot pays a fine for how far it drifts from a reference: a frozen copy of itself, saved just before this stage of training began.

"KL" is short for KL divergence, a number that measures how different two models' habits are, in other words how differently they would spread their bets over the next word. The further the chatbot's habits move from the reference, the bigger the fine. One dial, called beta, sets how heavy that fine is.

What the video shows

The video treats the penalty as a leash. Training can push an AI to cheat for points, so you tie it to where it started. But the leash has two ways to fail: too loose and the AI cheats anyway, too tight and nothing improves.

An everyday example

Imagine a new chef hired to win over a food critic who, it turns out, loves salt. Let the chef rewrite the house recipes freely and every dish becomes a salt bomb that scores beautifully and tastes awful. Forbid any change and the menu stays exactly as it was, so hiring the chef bought you nothing. The owner's real job is to decide how much change to allow, and nobody can tell them the right amount in advance.

What the course measured

The course turns the dial in a simulation and watches two numbers: the judge's score and the true quality of the answers.

| Beta (weight of the fine) | Judge's score | True quality | |---|---|---| | 0 | 31.7 | 0.000 | | 1 | 31.7 | 0.000 | | 5 | 28.3 | 0.285 | | 15 | 12.9 | 1.542 | | 60 | 9.6 | 1.769 | | Reference model, untouched | 9.2 | 1.791 |

Reading it from top to bottom:

• At 0 and 1 the leash does nothing. Cheating is worth so much more than the fine that the chatbot simply pays it and carries on.
• Between 5 and 15 the behaviour swings. In that stretch the score falls sharply and most of the quality comes back.
• At 60 quality returns, but it still sits a little below the untouched reference model.

So a loose leash gives you the cheating back, and a tight one gives you the starting model back.

A common misconception

People often say the KL penalty prevents reward hacking. The honest version is weaker. It limits how far the chatbot can travel from the reference, and because cheating requires travelling, that helps. What it never does is fix the judge that rewarded the cheating in the first place. It restrains the model; it does not correct it.

Why it matters

The useful range of the dial is narrow, its position changes with the judge, and the only way to find it is to measure. It also means the reference model has to be good, because a tight leash simply hands you the reference back. That is a large part of why this stage of training is difficult in practice, even though the idea is simple.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.