Pre-norm vs post-norm: the layer move that made warmup unnecessary

What pre-norm and post-norm mean

A transformer, the design behind today's chatbots, is a stack of blocks. Each block has two parts, attention and a feed-forward layer, and each part adds its result onto a running total that flows through the whole stack. That running total is the residual path. Think of it as a main road, with each part as a side road that branches off, does some work and merges back in.

Transformers also use LayerNorm, a step that resets a list of numbers to a steady scale so they do not drift too big or too small. The whole question is where to put it:

• Post-norm, the 2017 original: add the side road's result to the main road, then normalize the main road itself.
• Pre-norm, used by nearly every model built since 2019: normalize the copy that enters each side road, and never touch the main road.

What the video shows

The video shows the original 2017 transformer normalizing its signals right on the main path, and needing a slow warmup before it would train. Moving that step onto each side road removed the need for the warmup. Almost every model since 2019 does it that way.

An everyday example

Imagine a report passed along a line of twelve editors. In the post-norm office, after each editor adds their notes, someone reformats the entire report, so the original gets reshaped twelve times over. In the pre-norm office, each editor receives a neatly formatted copy to read, but their notes go onto the original, which nobody ever reformats. The second office keeps a clean line from the first draft to the last.

Why post-norm needed warmup

Training works by sending a correction signal, the gradient, backwards through every block to tell each setting which way to move. With post-norm, that signal starts out poorly balanced across the depth of the network. In the course's twelve-block model, before any training step, the signal at the last block was 1.65 times the size of the one at the first block. With pre-norm, the ratio was 1.32.

The usual fix was learning rate warmup: begin training with tiny steps and grow them gradually. The course trained the same model all four ways. Perplexity scores how surprised a model is by real text, and lower is better:

| Where LayerNorm sits | Warmup? | Final perplexity | |---|---|---| | Pre-norm | No | about 38 | | Pre-norm | Yes | about 38 | | Post-norm | Yes | about 38 | | Post-norm | No | 308 |

Without warmup, post-norm ends up roughly eight times worse. Add warmup and it catches up with pre-norm. Move the LayerNorm and the warmup is no longer needed at all.

How to read this

The lesson is bigger than one layer. It is tempting to file warmup under general good habits, but here it was a patch for one particular placement of the normalization. Change the placement and the patch becomes unnecessary. When a recipe contains a step nobody can explain, it is worth asking which other choice that step is compensating for.

It also changes how to read the famous 2017 transformer diagram. It is best treated as history rather than as a specification: modern models quietly moved this one step, and the diagram still shows the old position.

Why it matters

Small placement choices inside a model can decide whether training works at all. Here, the same ingredients in a slightly different order turned a fragile model that needed special handling into one that trains well without it, which is why pre-norm became the default.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.