Continuous batching: no more waiting for the slowest reply

What continuous batching is

AI servers save money by working on several requests in the same step, a group called a batch. Continuous batching changes how that group is managed. Instead of a fixed group that starts and finishes together, the server keeps a set of slots. The moment one request finishes, the next one waiting in the queue takes over that slot on the very next step. The server makes this decision after every single token (a word or a piece of a word), not once per request.

What the video shows

The old approach, known as static batching, made every request in a group wait for the slowest one before anybody got an answer. Refilling each free spot right away cut the average wait from 83 seconds to 8 seconds in the course's simulation.

An everyday example

Imagine a barbershop with eight chairs and a strange rule: new customers can only come in once all eight haircuts are finished, and nobody is allowed to leave early. A quick trim sits in the chair for an hour because the person next to them is getting a full colour treatment. Meanwhile a queue builds up outside.

Continuous batching is the normal barbershop. A chair frees up, the next customer sits down.

How it works

The trouble with fixed groups is that replies vary wildly in length. Some are twenty tokens, others five hundred. A fixed group keeps running until its longest reply is done, and the finished ones keep holding their slots, doing nothing useful, until then.

The course simulated 64 requests with realistic lengths on a server with eight slots. Half of the replies were 18 tokens or shorter, and the longest was 231:

| | Static groups of 8 | Continuous, 8 slots | |---|---|---| | Total time | 176.9 s | 109.0 s | | Tokens per second | 10.6 | 17.2 | | Average wait per request | 83.2 s | 8.1 s | | Wasted slot-steps | 3,214 | 0 |

A wasted slot-step is one slot spending one step on a request that has already finished. Output went up 1.6 times. The average wait dropped by more than ten times. The reason is simple: under the old rule, a request that was ready after four steps still had to sit there until a 231-token neighbour finished.

What it does not change

It is easy to read those numbers as proof that continuous batching escapes the usual batching trade-off, where more output for the server means longer waits for each user. It does not. Sharing a step with others still costs each request a little time. What continuous batching removes is waiting that was never necessary in the first place, which is why both numbers improved in this simulation.

Why it matters

The idea is as simple as it sounds, and the payoff is large. According to the course, every serving stack in production (the software that runs a model for many users at once) now works this way. It does leave one problem behind: requests coming and going leave the memory full of scattered gaps. That is what PagedAttention, the next idea in the series, deals with.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.