#174Dieses Video und der Text sind auf Englisch.
Group quantization: stop one extreme number from ruining millions
When millions of AI numbers share one rounding scale, a single extreme value wrecks them all. Giving each small group its own scale fixes most of it, cheaply.
Im Detail
What group quantization is
Quantization shrinks an AI model by rounding its numbers (its weights, the values it learned in training) so they fit in fewer bits. Rounding needs a scale: a kind of ruler that says how far apart the allowed values sit. Group quantization gives each small group of weights its own ruler, instead of making millions of weights share one. The course's version uses one scale for every 128 weights that sit next to each other.
What the video shows
The video shows how one number can ruin millions. When all of an AI's numbers share a single rounding scale, one extreme value stretches it so far that it becomes useless for everything else. Giving each small group its own scale fixes most of the damage, and it does so cheaply.
An everyday example
Imagine a bar chart of a year of spending, where one month includes buying a car. The axis stretches to fit the car, and every other month shrinks to a thin sliver. You can no longer tell a cheap month from an expensive one. Now draw one small chart per quarter, each with its own axis. Only the quarter with the car stays squashed; the other nine months are readable again.
That is the whole move. The extreme value still exists, but now it only distorts its own small group.
What the course measured
The course rounds a real half-billion-weight model to 4 bits, which allows just 16 values per number.
- In one weight table of about 4.3 million numbers, the biggest weight sits 24 standard deviations from the middle. A standard deviation is the usual spread of the values, and only about one weight in six thousand gets even six of them out. With one scale for the whole table, that single weight sets the step size, and the ordinary weights end up on only two or three distinct values.
- Model quality is scored with perplexity, a measure of how confused the model is by real text, where lower is better:
| One scale per... | Perplexity | |---|---| | whole table | about 365 million | | row of the table | 46 | | group of 128 weights | 31 |
For comparison, the original unrounded model scores about 23. A scale per row cut the rounding error by about 3.4 times, and groups of 128 cut it further still.
What it costs
Every scale has to be stored as well. One 16-bit scale for each group of 128 weights adds 16 divided by 128, or 0.125 bits, to every weight. So "4-bit" storage really takes about 4.125 bits per weight. That is roughly 3% more space to turn a useless model into a usable one.
Why it matters
At 4 bits, the bit count alone does not decide quality. What decides it is how many weights each scale has to serve. The smaller that territory, the less harm a single extreme value can do. More refined 4-bit formats, such as NF4, keep the same idea, with one scale per small block of weights rather than one per table.
The general lesson reaches beyond AI: whenever one shared setting has to fit everything, the most extreme case decides it for everyone. Splitting the work into smaller groups lets each group be treated on its own terms.
Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.
Auch auf
Mehr aus diesem Kapitel
#174
#175Prefill vs decode: why a chatbot pauses first, then streams smoothly
#176KV cache memory: when a chatbot's notes outgrow the model itself
#177Batching: more output per second, but a longer wait for each user
#178Continuous batching: no more waiting for the slowest reply
#179PagedAttention: the parking-lot trick behind busy AI servers
Text mit AI-Unterstützung aus Kapitel 13 unseres kostenlosen Kurses geschrieben.