Model parallelism: slicing an AI that is too big for one chip

What model parallelism is

Some AI models are simply too big for one chip. It is not only the model's own settings that need room: training also has to store the corrections for every setting and the optimiser's bookkeeping, the running averages the training algorithm keeps. Model parallelism is the family name for the answer. Instead of giving every chip a full copy of the model, you cut the model itself into pieces and spread the pieces across many chips.

The course describes two ways of cutting, tensor parallelism and pipeline parallelism, plus a close relative, context parallelism, which splits the text being read rather than the model.

What the video shows

The video starts from the problem: big AIs do not fit on one chip. Then it shows three ways to split the work. Each giant table of numbers is divided across several chips. Groups of layers are sent to different chips. And very long texts are split up too. It ends with a real training run that combined all of this on 16,384 GPUs.

An everyday example

Imagine a factory building one enormous vehicle. There are two natural ways to share the job:

• Split each big part. The chassis is too large for one welder, so four welders each take a section of it and work at the same moment. That is tensor parallelism: one giant table of numbers is divided so that several chips each compute their own share of it.
• Split the sequence of steps. Station one builds the frame, station two adds the engine, station three fits the interior, and the vehicle moves down the line. That is pipeline parallelism: each chip owns a run of consecutive layers and passes its result on to the next chip.

How the pieces fit

• Tensor parallelism divides individual weight matrices. A language model is mostly made of huge grids of numbers that get multiplied together, and this cuts each grid into slices that live on different chips.
• Pipeline parallelism hands each chip a block of layers in order, like stations on an assembly line.
• Context parallelism splits the text itself into chunks on different chips. It only becomes necessary once the texts are long enough for attention, the step where each word looks back at the words before it, to become the biggest cost.

These are combined with data parallelism, which copies the whole model instead of cutting it. Llama 3 used all four at once: its tables split 8 ways, its text split up to 16 ways, its layers divided into 16 groups, and up to 128 copies of that whole split-up arrangement running side by side, across 16,384 H100 GPUs, a high-end kind of AI chip.

Why it matters

At the frontier, the memory needed is far beyond any single chip. The same chapter works out that training a 70-billion-parameter model needs about 1.12 terabytes just for its settings, their corrections and the optimiser's bookkeeping. That is fourteen 80 GB GPUs' worth before counting any of the in-between results. Without cutting the model up, a run like that could not even start.

The course deliberately keeps this topic short and points readers to Stanford's CS336 lectures for the engineering details. What it keeps is a single number that turns all this engineering into time and money: the share of the chips' peak arithmetic a real run actually achieves.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.