Multi-Token Decoding

The single biggest reason chatbot responses take seconds instead of milliseconds is that they generate one token at a time, in order — a 500-token answer means 500 sequential steps. Multi-token decoding breaks that bottleneck, and understanding it is the key to making an LLM-powered product feel fast. You will use it directly when you build any real-time AI application: a customer-support bot, a coding assistant, a voice assistant that must reply instantly. It also underlies how reasoning models plan ahead and how on-device systems stay responsive on small hardware. This concept unlocks practical engineering of inference latency: you will understand what a forward pass is, why it is parallel and why generation is not, and how a draft model and a verification pass can give you the same answer in half the time. Later topics like Quality-Cost-Latency Tradeoff and Temperature and Sampling will build on this picture.

When you ask a large language model a question, it does not write out its whole answer at once. It produces one small piece — a token, often a word or part of a word — then takes that piece as part of its input to produce the next one, and so on. Each of those steps is called a forward pass. The model looks at the tokens so far and computes a probability for every possible next token, then picks one (usually the most likely). That picked token is added to the sequence, and the whole process repeats. This is sequential: step two cannot start until step one is finished, because the model needs the first token to know what to compute next. There is no way to peek ahead, because the answer is generated, not retrieved. Multi-token decoding is a set of techniques that try to do the same work in fewer passes. The clever trick is: instead of asking the big, slow model to produce one token at a time, you let a small, fast 'draft' model guess several future tokens in advance, and then ask the big model to check those guesses all at once. The big model can verify several guesses in a single forward pass (because checking is parallel — it can score many candidates at once). If the guesses are correct, you are done after one verification, having produced several tokens in one step. If some are wrong, you discard the wrong ones and start over from the last correct guess. The result is exactly the same output you would have gotten the slow way — the big model is the authority, and the draft is just a guess — but you have used fewer sequential passes.

Imagine you are a very thorough, very slow proofreader. Your job is to correct a long manuscript, but you can only look at one word at a time, and after each word you must decide the next word. That is like standard generation. Now imagine you have a quick assistant who is not as accurate but is much faster. The assistant writes down a whole sentence — a guess at what you would write. You then scan the entire sentence at once — because you can check many words in parallel — and you either approve the whole thing or stop at the first mistake. If the assistant is usually right, you save a lot of effort: you correct a whole sentence in one pass instead of word by word. The key is that you are the final authority: if the assistant writes 'the cat sat on the mat' and you agree with every word, you have just produced five tokens in one go. If the assistant writes 'the cat sat on the dog' and you disagree with 'dog', you reject that and everything after it, and you start again from 'on'. The output is identical to what you would have produced on your own, because you are the one who approves every token. The analogy breaks down in one important way: the assistant and the proofreader are not two people — they are two different models. The small draft model is not 'dumb'; it is trained on the same kind of data, just with fewer parameters, so it is faster but less accurate. Also, the draft is not a guess at what you would write from nothing — it reads the same prompt and the tokens you have already approved, and tries to continue from there. Finally, the verification is not a separate check: in practice, the big model computes probabilities for all next tokens at once, and if the draft’s token is the most likely (or within an acceptable range), it is accepted. The maths of that decision is where the accuracy guarantee comes from — if the big model would have picked that token anyway, accepting it does not change the output.

Multi-token decoding is a family of inference-speed techniques that generate more than one token per sequential forward pass, typically by using a small, fast model to propose a draft of several future tokens which a larger, slower model then verifies in a single parallel pass, accepting the draft up to the first disagreement so that the final output is identical to what the large model would have produced one token at a time.

This concept sits inside Large Language Models. Your notes already cover Tokenization, the Context Window, and Temperature and Sampling — all of which appear in the story. Tokenization defines what a token is: the small pieces the model reads and writes. A forward pass reads a sequence of token IDs and produces a probability distribution over the vocabulary. Context Window bounds how many tokens the model can attend to — relevant because the draft and verification passes both operate within that window. Temperature and Sampling decide how the next token is picked from the distribution; multi-token decoding works with any such policy, though the verification rule must match it to preserve exact equivalence. It also connects to the Quality-Cost-Latency Tradeoff: you are spending a bit more compute (running the draft model) to reduce latency (wall-clock time the user waits). And to Reasoning Models: those extend generation to produce long chains of thought, which make the sequential bottleneck worse, so multi-token decoding is especially useful there. Finally, it builds on Foundation Models: the large model is the fixed authority you want to keep; the draft is a cheap imitator you can train or even use the same model in a smaller form.