#174هذا الفيديو ونصه باللغة الإنجليزية.
Speculative decoding: a small AI guesses, the big AI checks
Speculative decoding lets a small AI guess the next words and a big one check them all at once. Learn why the text stays the same, and when it slows things down.
بالتفصيل
What speculative decoding is
A language model writes one token at a time. A token is a small piece of text, often a word or part of a word. For a big model, every new token means one full pass through the whole model, and that is slow. Speculative decoding is a trick to speed this up without changing a single word of what the model writes.
The idea: a small, cheap model races ahead and guesses the next few tokens. The big model then checks all of those guesses together, in one pass. Checking several tokens in one go costs barely more than producing one, so every guess the big model agrees with is time saved.
What the video shows
The video shows the partnership: a small AI guesses the next few words, and the big AI checks them all at once. The final text is exactly what the big model would have written on its own. And it points out the catch: this only makes things faster when the guesses are usually right.
An everyday example
Imagine your phone keyboard suggesting the next four words of your message. If all four are exactly what you meant to type, one tap saves you four words of typing. If the first two are right and the third is wrong, you keep the first two, type your own third word, and ignore the rest. But if the suggestions are almost always wrong, reading them only slows you down.
That is speculative decoding, with the big model in your place.
How it works
- Draft. The small model proposes a handful of tokens, one after another.
- Verify. The big model runs once over all of them and works out what it would have written at each position.
- Accept. The longest run of matching guesses is kept. At the first mismatch, the big model's own token is used instead, and that one comes free from the same pass. The rest is thrown away and the cycle starts again.
Because a guess only survives when the big model agrees with it, the output is the big model's own. When the model picks words with some randomness, a special acceptance rule is needed, and it has been proved to give exactly the same odds as the big model working alone.
When it helps, and when it hurts
Everything depends on the acceptance rate: how often the big model agrees with the guesses. Here is the speed-up the course calculates for drafts of 4 tokens, with a small model that costs a fifth of the big one:
| Acceptance rate | Speed-up | |---|---| | 30% | 0.79x, so slower | | 50% | 1.08x | | 70% | 1.54x | | 90% | 2.28x |
With an even cheaper drafter, 90% acceptance reaches roughly three times the speed. At 30%, you pay for all the drafting and keep very little of it. Longer drafts are a trap too: with 8-token drafts and a drafter costing a tenth of the big model, you get 3.40x at 90% acceptance and 0.79x at 30%. Same setup, a win or a loss.
A common misconception
It is easy to think speculative decoding always makes a model faster. It does not. It never changes the answer, but its speed depends on how well the small model predicts the big one, and that is a number you have to measure on your own traffic.
Why it matters
Rounding a model to fewer digits, one popular way to make AI cheaper, trades away a little quality. Speculative decoding leaves the output exactly as it was. But it is a bet on a measured number, not a free lunch.
Learn it step by step in Chapter 13 of our free course AI From Scratch: Making Inference Cheap: KV Cache, Batching and Quantization.
متاح أيضًا على
المزيد من هذا الفصل
#174
#175Prefill vs decode: why a chatbot pauses first, then streams smoothly
#176KV cache memory: when a chatbot's notes outgrow the model itself
#177Batching: more output per second, but a longer wait for each user
#178Continuous batching: no more waiting for the slowest reply
#179PagedAttention: the parking-lot trick behind busy AI servers
كُتب النص بمساعدة الذكاء الاصطناعي استنادًا إلى الفصل 13 من دورتنا المجانية.