Feed-forward layer: the two thirds of an AI block nobody mentions

What the feed-forward layer is

The AI models behind chatbots are built from one unit repeated many times, called a transformer block. Each block does two jobs, one after the other. First, attention lets every word gather information from the words around it. Then a feed-forward layer, a small neural network, takes each word on its own and works on what that word has just gathered. During this second step, no word talks to any other word.

Diagrams and explainers rarely spend much time on this part. Yet it holds about two thirds of the parameters in every block: the adjustable numbers a model tunes while it learns.

What the video shows

The video walks through the two halves of a transformer block. Attention moves information between words. The feed-forward layer then processes each word alone. And when you count what each half contains, the quiet half turns out to be the big one: two thirds of the block, whatever the size of the model.

An everyday example

Imagine a study group before an exam. In the first half of the session, everyone swaps notes: who knows what, who is missing which piece. That is attention. In the second half, each student goes to a separate desk and works through what they collected, using everything they already know. That is the feed-forward layer.

Swapping notes is essential, but it only moves notes from one person to another. The working-through happens at the desks, and that is also where most of the knowledge sits.

How it works

Inside the model, every word (strictly, every token: a word or a piece of one) travels as a long list of numbers. The feed-forward layer does three things to that list:

1. Widen it. The list is stretched to four times its length, which gives the network room to work.
2. Bend it. A function called GELU, a smooth on-off switch, lets some values through and quiets others.
3. Shrink it back. The list returns to its original length, ready for the next block.

Why is this needed at all? When attention passes information along, it only weights and adds lists of numbers together. That is straight-line math, and stacking straight-line steps only ever gives you another straight-line step. The bend in the middle of the feed-forward layer is what lets the model compute something richer inside each word.

Where the parameters sit

The course counts the parameters in a single block at three widths:

| Block width | In attention | In the feed-forward layer | |---|---|---| | 128 | about 66 thousand | about 132 thousand | | 768 | about 2.4 million | about 4.7 million | | 4,096 | about 67 million | about 134 million |

The split is two thirds to one third every time. The reason is simple arithmetic: attention uses the equivalent of four square tables of numbers, while the feed-forward layer, with its widen-and-shrink shape, uses the equivalent of eight.

It also earns its size. In the course's small experiment, adding a feed-forward layer to a model that only had attention improved perplexity, a score of how surprised a model is by real text where lower is better, from 43.2 to 39.9.

Why it matters

When you hear that a model has billions of parameters, remember that inside each block, most of them sit in this per-word network, not in the attention machinery that gets all the headlines. So most of what the model has stored from training lives here. That is also why mixture of experts, a trick for growing a model without making each word more expensive to process, targets exactly this part.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.