KV cache: how an AI writes nine times faster without cutting corners

What the KV cache is

A language model writes one token at a time. A token is a word or a piece of a word. To choose each new token, the model looks back over everything it has written so far. For every earlier token it builds two small lists of numbers: a key, which works like a label saying what that token is about, and a value, the information it hands over when a later token looks it up.

The KV cache (K for keys, V for values) simply keeps those lists in memory. They are built once and reused, instead of being rebuilt from scratch at every step.

What the video shows

The video compares two ways of producing the same AI answer: with the cache it takes about 9 seconds, without it about 79, and the words come out identical. In the course's test behind it, the answer is 48 tokens long and the exact times are 8.85 s and 78.95 s. Nothing was lost. Only repeated work was removed, and the answer arrived roughly nine times sooner.

An everyday example

Imagine two cashiers adding up a long shopping list. The first keeps a running total and adds each new item to it. The second re-adds every item from the top each time a new one is scanned. Both reach the same total. The second one just gets slower and slower as the list grows.

A model without a cache is the second cashier.

Why the old work never changes

The trick depends on a rule built into these models: a token can only look backwards, never forwards. Experts call this the causal mask. So when token number 8 arrives, token number 7 cannot see it, and its key and value stay exactly as they were. Recomputing them gives the same numbers again, so storing them is pure savings.

With the cache, each step feeds the model just the one new token. That token compares itself against all the stored keys and collects what it needs from the stored values.

How big the saving gets

The gap grows with the length of the text. Without a cache, the work for each new token grows with the square of the context, meaning the text so far: double the length and each step costs roughly four times as much. With the cache, it grows only in step with the length. In the course's count, producing 2,048 tokens without a cache takes over 1,000 times more arithmetic.

Skipping the cache does not even save memory. In a separate test on a laptop graphics card, generating 256 tokens without the cache needed 8.3 times more working memory, all spent to produce the same tokens more slowly.

A common misconception

The word cache often sounds like a shortcut that trades quality for speed. Not here. The KV cache is exact: it deletes arithmetic that was always going to give the same result. The course calls it the most consequential speed-up in serving language models.

It does have a price, and that price is memory. The notes have to live somewhere for as long as the conversation lasts, which is a story of its own later in this chapter.

Why it matters

Every chatbot reply is written token by token. Without this cache, long answers would crawl, and each new word would cost more than the one before it. With it, every step handles a single new token and reuses everything else.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.