Prefill vs decode: why a chatbot pauses first, then streams smoothly

What prefill and decode are

When you send a message to a chatbot, the model works in two phases.

• Prefill is the reading phase. The model takes your whole prompt and handles all of its tokens together, in a single pass. A token is a word or a piece of a word.
• Decode is the writing phase. The model produces the reply one token at a time, running a full pass for every single token.

Same model, same hardware, same weights (the numbers the model learned during training). Yet the two phases behave in completely different ways.

What the video shows

The video contrasts the two phases. Reading your prompt is the cheap part, per token. Writing the reply is about six times slower per token. That split is why a long chat starts slowly and then streams at an even pace.

The numbers come from the course's test: reading a 40-token prompt cost about 25.6 milliseconds per token, while each token of the reply cost about 166 milliseconds.

An everyday example

Paste a long report into a chatbot and ask for a summary. There is a pause before the first word appears. That pause is mostly prefill, and it grows with the size of what you pasted. Then the words arrive at a steady rhythm. That rhythm is decode, and it stays about the same whether the conversation so far is short or long.

How it works

The difference comes from moving data, not from doing maths. For any pass at all, the model has to pull every one of its weights out of memory.

• In prefill, one trip through the weights serves every token of the prompt at once, often hundreds of them. There is plenty of calculation per trip, which is exactly the kind of work a graphics chip is built for. The limit is raw computing power.
• In decode, the same full trip serves one single token. There is almost no calculation per trip. The limit is how fast data can flow out of memory, known as memory bandwidth.

Imagine a cook who has to carry every pot and pan from the storeroom before cooking anything. Cook forty dishes after one trip, and each dish is cheap. Cook one dish per trip, and every dish pays the full carrying cost.

The course measured a detail that surprises most people: in decode, going from 16 to 1,024 earlier tokens in the conversation made no measurable difference to the time per step. Looking back over the history is real work, but it is tiny next to the fixed cost of hauling the weights.

A common misconception

Most people expect reading the prompt to be the slow part. Per token, it is the cheap part. And the smooth streaming you see is not a display effect added by the app. It is simply the pace of decode.

Why it matters

These two phases are behind the two speed numbers that AI services report:

| Number | Set by | What it depends on | |---|---|---| | Time to first token | Prefill | Grows with the length of your prompt | | Tokens per second | Decode | Stays roughly constant |

Knowing which phase you are waiting on tells you what to expect. The fixed cost of decode is also the reason AI servers group several users into the same step, the idea behind batching.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.