Training memory: 16 bytes per setting, plus a mountain of scratch work

What training memory is

Training memory is the room a computer chip needs while it is teaching an AI model, as opposed to the room the finished model takes up. The two are very different. Counting compute tells you how long a training run will take. Counting memory tells you whether it can begin at all: if the numbers do not fit on the chip, nothing starts.

What the video shows

The video gives two numbers that surprise most people. First, training stores four numbers for every setting of the model, so a model with 70 billion settings needs 1.12 terabytes before any work starts. Second, the scratch work produced along the way can weigh 52 times more than the model itself.

Where the four numbers come from

A model's parameters are its adjustable numbers, its "settings". In plain training, with every number stored at full detail (4 bytes each), each setting drags three companions along:

1. The weight itself, the setting's current value.
2. The gradient, the correction just calculated for it: which way to nudge it, and by how much.
3. and 4. Two running averages kept by the adjustment method (the course uses AdamW). One tracks the usual direction of recent corrections; the other tracks how large they tend to be.

Four numbers at 4 bytes each is 16 bytes per setting. For 70 billion settings that is 1.12 terabytes, the capacity of fourteen large 80 GB accelerator chips, and not a single step of training has happened yet. On a laptop graphics card, the course checked this rule against real measurements and found them within 3% of each other.

The scratch work

Then come the activations: the intermediate results of the model's forward pass (its guess), which have to be kept until the backward pass uses them to work out the corrections. They grow with the batch, the number of text examples processed together in one step.

In the course's measurements, a small model with 5.8 million settings needs 89 MB of permanent state. Its activations take about 290 MB with a batch of 8, about 1.2 GB with a batch of 32 and about 4.7 GB with a batch of 128. That last figure is the 52 times in the video.

A sizable part of it, about 512 MB in that run, is not the model's inner layers at all. It is the output scores: for every token (a small piece of text), the model produces one score for every entry in its vocabulary, and all of them sit in memory.

An everyday example

Imagine a chef with a small recipe card. The card is the model. To cook for a banquet, though, the chef needs counter space for every bowl of half-chopped ingredients until each dish is finished. Double the guests and you need double the counter. The recipe did not change; the scratch work did.

Why it matters

Memory, not just speed, decides what can be trained on a given machine and how many chips a run needs. Which part dominates depends on the shape of the run: the 16 bytes per setting for huge models, the scratch work for big batches and big vocabularies. One catch: storing numbers with less detail does not simply shrink the 16 bytes. The usual recipe still keeps a full-detail copy of every setting next to the rougher ones, and the total comes back to 16 bytes per setting. Spreading the load across many chips is a story of its own.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.