Constitutional AI: training a model with a written rulebook

What Constitutional AI is

To turn a raw language model into a helpful assistant, trainers collect preferences: pairs of answers where someone has marked the better one. Usually that someone is a person. Constitutional AI hands the job to a model instead. The model receives a written set of principles, the "constitution", and uses it to judge answers. The wider family of methods that do this is called RLAIF, short for reinforcement learning from AI feedback, as opposed to feedback from humans.

What the video shows

The video shows an AI that trains another AI with a rulebook. Instead of people picking answers, an AI checks them against a written list of principles. It is much faster, and it works because judging is easier than writing. In practice the judging model does not even have to be a different one: a model can be asked to review its own answers.

An everyday example

Imagine a new reporter handed the newspaper's style guide: be accurate, be fair, name your sources, keep it clear. Before filing a story, they reread their own draft against each rule, mark what falls short and rewrite it. Nobody else in the newsroom had to read the draft. Now imagine doing that for an enormous pile of drafts, at machine speed and machine cost.

How it works

A model guided by the principles can play the judge in two ways:

• Critique and revise. It reads one of its own answers, points out where it breaks a principle, and writes an improved version.
• Pick between two. It reads two candidate answers and chooses the one that follows the principles better.

Either way, the output is preference data, the same kind of material people normally produce by comparing answers by hand. From there, training carries on as usual. The difference is cost and speed. Data that would otherwise need a person to read and compare every pair can be produced at machine speed and machine cost.

A common misconception

The obvious objection is that the model is marking its own homework. The course takes that seriously: the concern is real. But the method holds up better than you might expect, for one reason. Judging is easier than generating. Most people could not write a hit song, yet they can tell which of two songs is catchier. The same gap lets a model pick out the better answer more reliably than it could produce that answer from scratch.

It is also the same gap the human version depends on. Preference training with people relies on it too: they can say which of two answers is better even when they could not have written either one.

Why it matters

Human feedback is slow and expensive, which limits how much of it anyone can gather. Swapping in a model guided by written principles turns that bottleneck into a job a computer can do quickly and cheaply, and it puts the rules in writing, where the people training the model can read and change them.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.