Causal mask: the triangle that keeps a language model from cheating

What a causal mask is

A language model is trained on one task: read the words so far and guess the next one. During training, though, the whole sentence sits in memory at once. Something has to stop word 4 from simply glancing at word 5, which is exactly the answer it is being graded on. That something is the causal mask: a rule that lets each position look only at itself and at the positions before it, never at the ones after. "Causal" just means "in the order things happen": the past can shape the present, the future cannot.

What the video shows

The video starts from the goal: an AI learns by guessing the next word, so it must never see that word. Then it shows the fix, a triangle-shaped block that lets each word look only at the words before it, never after.

An everyday example

Imagine a teacher running a story quiz. The whole story is printed on one sheet, and students must guess each line before reading it. So she covers the sheet with a card cut like a staircase: on step 1 you can see line 1, on step 2 lines 1 and 2, and so on. The card never shows a line you have not reached yet.

Now draw that card as a grid, with one row per line of the story and one column per line it might look at. The visible part is a triangle. That is the causal mask.

How it works

Inside a language model, each word builds its understanding by mixing in information from other words, and a grid of weights says how much each word takes from each other word. The mask blanks out every cell to the right of the diagonal, which is the future.

The simplest version gives every allowed word the same weight:

| Word | Can look at | Weight on each | |---|---|---| | 1 | word 1 | 1 | | 2 | words 1 and 2 | 1/2 | | 3 | words 1 to 3 | 1/3 | | 4 | words 1 to 4 | 1/4 |

Every row adds up to one, and the upper-right corner is all zeros. The course checks that doing this as one big multiplication of number grids gives the same result as a slow word-by-word loop, down to tiny rounding differences.

In practice the mask is applied with a neat trick. Before the model turns its raw scores into weights, using the softmax step that converts scores into shares adding up to 100%, every forbidden score is set to minus infinity. Softmax turns minus infinity into exactly zero and shares out all the weight among the allowed words. Hiding the future and making the weights add up happen in a single move.

Why minus infinity and not just a very large negative number? The course gives three reasons: it says exactly what the mask means, it survives being stored in the smaller 16-bit number format that models often use, and it spares you from wondering whether your "large" number is large enough.

Why it matters

Without the mask, the model would score brilliantly in training by copying the next word, then be useless when it writes text for real, where there is no next word to copy. It is the same trap as target leakage, where an input secretly contains the answer, except that here the leak would be built into the model's own wiring.

The triangle also sets up the rest of attention. Once the future is hidden, the only open question is which scores to put in the allowed cells. Equal scores give a plain average. Learned scores, different for every sentence, give attention.

Learn it step by step in Chapter 9 of our free course AI From Scratch: Attention and the Transformer Block, Derived From an Average.