KV cache memory: when a chatbot's notes outgrow the model itself

What KV cache memory is

To write quickly, a language model keeps notes on every token in the conversation. A token is a word or a piece of a word. These notes are called the KV cache: for each token the model stores a key and a value, small lists of numbers that later tokens look up, so it never has to rebuild them. That saves a huge amount of repeated work. The price is memory, and that price is what this concept is about.

What the video shows

The video takes a model in the 7B class, meaning it has about 7 billion learned numbers. In a 128,000-token conversation, one user's notes fill 15.62 GB. The model's own weights, the numbers it learned in training, take 13.0 GB. The notes have outgrown the AI. The video also points to the remedy: when parts of the model share their notes, the pile shrinks.

An everyday example

Imagine an interpreter who writes an index card for every word said in a meeting, so nobody ever has to repeat themselves. For a short meeting, it is a small stack. For a very long one, the stack of cards can end up bigger than the dictionary the interpreter brought. The cards are useful, but somebody has to find room for them.

How to work out the size

The size follows a simple recipe. For each token, multiply together:

• 2, because there is one key and one value;
• the number of layers, the stacked processing stages inside the model;
• the number of key-value heads, meaning how many separate sets of notes each layer keeps;
• the length of each note;
• the bytes used to store each number.

For the small model tested in the course, that gives 12,288 bytes per token. The course checked the recipe against what the memory allocator (the part of the software that hands out memory) really used, and it matched exactly. For the 7B-class design the course uses as its example (32 layers, 8 sets of notes per layer, each note 128 numbers long), the same recipe gives 128 KB per token, which is where the 15.62 GB comes from.

One more surprise: only the total number of tokens matters. Whether 32 people hold 2,048 tokens apiece, 64 people hold 1,024, or 128 people hold 512, the cache came to the same 768 MB, because every one of those cases adds up to 65,536 tokens.

How sharing shrinks the notes

In the classic design, every reader inside a layer (called a head) keeps its own set of notes. Two later designs cut that down:

• Multi-query attention: all readers share one single set.
• Grouped-query attention: a few sets, each shared by a group of readers.

The model in the course has 14 readers but only 2 note sets. If every reader kept its own, the cache would be seven times bigger: 84 KB per token instead of 12 KB. According to the course, the grouped version won out because sharing just one set cost real quality, while sharing within groups did not. Neither design saves any calculation. They exist only to shrink this memory bill.

Why it matters

A long conversation is not just a slower one. Its notes occupy a fixed slice of the machine for as long as the request is running. Once conversations get long, that memory becomes the real limit on what a server can do, and it is a big part of what long context windows cost.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.