#174Ez a videó és a szövege angol nyelvű.
Distillation: how a small AI learns from a big AI's doubts
Distillation trains a small AI to copy a big one, including how likely it found every option. See why those near misses teach more than the right answer alone.
Részletesen
What distillation is
Distillation is a way to get a smaller, cheaper AI model by training it to imitate a bigger one. The big model is called the teacher and the small one the student. The idea is older than it sounds: it dates from 2006 and was made popular by Geoffrey Hinton in 2015.
It is a different route from quantization, which keeps the same model and simply stores its numbers with fewer digits. Distillation builds a new, smaller model that learns to behave like the big one.
What the video shows
The video makes one point: a small AI can learn by imitating a big one, and the secret is to copy more than the big model's final answer. The student also copies how likely the teacher found every alternative, and those odds carry extra information.
An everyday example
Imagine studying for a multiple-choice exam with help from two friends. The first only hands you the answer key: B, D, A. The second, who aced the course, walks you through it: "B, but C was nearly right; A was never an option." Both give you the same answers. Only the second teaches you how the questions actually work.
The original version of this argument used handwritten digits. A picture can be a 7 that looks quite a bit like a 1. Knowing that resemblance is useful, and a plain answer key throws it away.
How it works
When a language model reads a phrase, it does not just pick the next word. It gives every possible word a probability. Here is what the model used in the course gives for "She poured the milk into the":
| Next word | Probability | |---|---| | jug | 13.6% | | cup | 10.5% | | bowl | 6.1% | | large | 3.8% | | milk | 3.6% |
A hard label is only the winner: "jug". A soft label is the whole list. It says jug is the favourite, cup is almost as good, bowl is plausible, and even "large" is still in play, which would lead the sentence somewhere else entirely, since it is a describing word rather than a container. A student trained on the soft label learns all of that at once.
There is one more dial, called temperature. Before its raw scores become probabilities, the teacher divides them by a number, T. A higher T flattens the list, so the runners-up count for more in the score the student is trained to improve. For the milk phrase, the top word is 2.24 times as likely as the third at T = 1, but only 1.50 times at T = 2. The order stays the same; the near misses just get a louder voice.
A common misconception
It is natural to think a student model just learns the teacher's answers. If that were all, you could skip the teacher and train on the correct answers directly. What the teacher really adds is its full spread of confidence: which wrong answers were close and which were far off.
Why it matters
A hard label tells the student what the teacher decided. A soft label also tells it how sure the teacher was, and which mistakes would have been reasonable. That extra information is why a small model can learn so much from a big one: it inherits the teacher's doubts, not only its verdicts.
Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.
Itt is
Továbbiak ebből a fejezetből
#174
#175Prefill vs decode: why a chatbot pauses first, then streams smoothly
#176KV cache memory: when a chatbot's notes outgrow the model itself
#177Batching: more output per second, but a longer wait for each user
#178Continuous batching: no more waiting for the slowest reply
#179PagedAttention: the parking-lot trick behind busy AI servers
A szöveg AI-segítséggel készült az ingyenes kurzusunk 13. fejezete alapján.