Alignment tax: the price an AI pays for being careful and polite

What the alignment tax is

After a language model has read its huge pile of text, it goes through alignment training: the stage that teaches it to be helpful, safe and well-mannered. That stage has a price. Models often come out measurably worse at some benchmark tasks, the standard tests people use to measure and compare models. That drop in raw ability is what people call the alignment tax.

The reason is simple once you see it. Alignment changes what the model is being rewarded for. A safe, hedged (wrapped in caveats) and neatly formatted reply is not always the most accurate reply. The two can be the same answer, but not always, and a test that only checks accuracy notices when they differ.

What the video shows

The video calls it the hidden cost of a well-behaved AI. Teaching an AI to be safe and careful can make it a little worse on some tests, because the careful answer is not always the most accurate one. And it adds a second point that is easy to miss: people decide what "well-behaved" means.

An everyday example

Imagine a sharp new employee who has just finished the company's customer-service training. They are now polite, they never promise what they cannot deliver, and they add the right warnings. Ask them a blunt technical question, though, and sometimes you get "it depends, you may want to check with a specialist" when they actually knew the exact answer. The training made them safer to put in front of customers and, on that one question, less useful. Both things are true at once.

Part fixable, part real trade-off

Not all of the gap is permanent. Some of it has been engineered away as training methods improved. But some of it is a genuine trade: an assistant that declines certain requests and flags its uncertainty will sometimes do worse on a test that rewards blunt accuracy. That remaining part is better seen as a trade-off to be managed than as a bug that will one day disappear.

Aligned with whom?

The word "aligned" hides a question. Follow the chain of who shapes the final model:

1. A company writes guidelines.
2. Hired contractors read those guidelines and compare pairs of answers, picking the better one.
3. Their choices train a reward model, a second model that learns to score answers the way those contractors did.
4. That scorer steers the model you actually talk to.
5. You ask it a question, never having seen any of the steps above.

Every link in that chain is a decision made by particular people. The training maths has no view on whether those decisions were good ones.

This is why two leading models can say no to different requests, why a single model can change its answers from one version to the next, and why "aligned" describes a process someone followed rather than a fixed property the model has.

Why it matters

When you read that a model is "aligned", ask two questions. What did it cost in plain ability, and on which tasks? And whose guidelines was it aligned to? The maths behind alignment training is settled. The human choices fed into it are not.

Learn it step by step in Chapter 11 of our free course AI From Scratch: From Base Model to Assistant: SFT, RLHF, DPO and GRPO.