Hoppa till innehållet

Den här videon och dess text är på engelska.

NF4: putting the rounding steps where an AI's numbers really are

AI Concepts #182

With only 16 values to round into, even spacing wastes many of them. See how NF4 packs them near zero, where an AI's numbers sit, and cuts the error by 28%.

I detalj

What NF4 is

When an AI model is squeezed down to 4 bits, each of its numbers (its weights, the values it learned in training) must be rounded to one of just 16 allowed values. The usual approach spaces those 16 values evenly, like the marks on a ruler. NF4, short for 4-bit NormalFloat, spaces them unevenly on purpose: many values close to zero, where most weights are, and only a few far out, where hardly any weights live. It comes from the research paper that introduced QLoRA.

What the video shows

The video makes one point clearly. With only 16 levels to round into, spacing them evenly wastes many of them. NF4 packs them near zero, where most of an AI's numbers sit, and in the course's test that cut the rounding error by 28%.

An everyday example

Imagine a shoe shop with room for only 16 sizes, where every customer leaves with the closest size in stock. If the shop stocks sizes in equal steps from the tiniest to the biggest, several shelves hold sizes almost nobody wears, and the gaps in the busy middle are so wide that most people get a loose fit. A sensible shop stocks most of its 16 sizes close together around the common feet and keeps only one or two for the extremes. Same 16 sizes, far more people in shoes that fit.

How it works

Inside a small block of weights, the values are shaped roughly like a bell curve, which statisticians call a normal distribution: lots of values near zero, fewer and fewer as you move away. NF4 is built around that shape. It places its 16 levels at the quantiles of a bell curve, meaning points chosen so that each level serves roughly the same share of weights. The result is tight spacing near zero and wide spacing in the tails.

Like group quantization, NF4 also gives each small block its own scale, so one extreme value cannot spoil everything. Here the block is 64 weights, which costs about 4.25 bits per weight instead of 4.

What the course measured

On the real weights of a half-billion-weight model:

  • Compared with evenly spaced 4-bit levels and one scale per 128 weights, NF4's rounding error was 28% lower.
  • Perplexity, a score of how confused the model is by real text where lower is better, was 24.55 with NF4 against 23.08 for the original full-precision model.
  • The NF4 version agreed with the original's predictions on 84.7% of tokens, the word pieces a model reads and writes.

An honest caveat

The 28% gain has two causes mixed together. In this test NF4 used smaller blocks (64 weights instead of 128) as well as smarter level spacing, and the course points out that telling the two effects apart would need an extra experiment. So the fair summary is this: better spacing plus smaller blocks, together, gave a clearly better result for a small extra cost in storage.

Why it matters

NF4 shows that the bit count is only half the story. Where you put the few values you can afford matters just as much. It also shows the price of 4 bits honestly: close to the original, but not identical, since roughly 15 tokens in every 100 came out differently.

Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.

Finns även på

Mer från det här kapitlet

Text skriven med AI-hjälp från kapitel 13 i vår gratiskurs.