Data parallelism: copy the AI to every chip, split the reading

What data parallelism is

Training a large AI model means showing it an enormous amount of text and correcting it over and over. That is far too much work for a single chip. Data parallelism is the standard way to share it out: every chip gets its own complete copy of the model, each copy reads a different slice of the text, and then the chips pool what they learned so that all the copies stay identical.

It is the default way to spread training across many chips, and usually the first kind of splitting people reach for.

What the video shows

The video shows three moves. A full copy of the AI goes onto every chip. Each chip is handed a different slice of text. Then the chips average what they learned, and training carries on. It presents this as the default way to scale training up.

An everyday example

Imagine a bakery perfecting a cookie recipe. Four tasters each get an identical copy of the recipe and a different tray of cookies. Each one writes down suggestions: a little less sugar, a bit more salt, two minutes less in the oven. Then they meet, average their suggestions, and all four update their copies in exactly the same way. Next round: new trays, same routine.

No taster had to try every cookie, yet the recipe improves as if one person had tasted them all. That is data parallelism.

How it works

1. Copy. Every chip (a GPU, the kind of processor used to train AI) holds the complete model.
2. Split. Each training step works on a batch, a group of examples. The batch is cut into slices, one per chip.
3. Learn locally. Each chip compares its copy's guesses with its own slice and works out corrections, called gradients: which way to nudge each of the model's settings, and by how much.
4. Average. The chips share their corrections and average them.
5. Update together. Every copy applies the same averaged correction, so all the copies are still identical at the start of the next step.

The averaging is the key detail. Without it, each copy would drift off in its own direction and you would end up with several different models instead of one well-trained one.

Where it hits a wall

Because every chip holds the entire model, the model has to fit on a single chip. There is also some waste. Each chip keeps its own identical copy of the optimiser's bookkeeping, the running averages the training algorithm keeps for every setting. A method called ZeRO improves data parallelism by refusing to store those redundant copies on every chip.

When the model itself is too big for one chip, copying it is not an option and it has to be cut into pieces instead. That is model parallelism, a separate idea.

Why it matters

Data parallelism is how adding hardware turns into faster training: more copies means more text read at the same moment, while the result is still one single model. At the largest scale it is combined with other kinds of splitting. Llama 3, for example, ran up to 128 copies of its model side by side. Each copy was itself spread over many chips using three other kinds of splitting, and the whole run used 16,384 H100 GPUs, a high-end AI chip.

Learn it step by step in Chapter 10 of our free course AI From Scratch: Pretraining an LLM: Data, Compute, Scaling Laws and Cost.