Multi-Token Decoding

The single biggest reason chatbot responses take seconds instead of milliseconds is that they generate one word or part of a word at a time, in order — a 500-token answer means 500 sequential steps. Multi-token decoding breaks that bottleneck, and understanding it is the key to making an AI-powered product feel fast. You will use it directly when you build any real-time AI application: a customer-support bot, a coding assistant, or a voice assistant that must reply instantly. It also underlies how reasoning models plan ahead and how systems stay responsive on small hardware. This concept unlocks practical engineering of speed: you will understand what a single step of processing is, why some steps can happen at the same time while others cannot, and how using a smaller, faster model to guess several future words can give you the same answer in half the time.

When you ask a large language model a question, it does not write out its whole answer at once. It produces one small piece — a token, often a word or part of a word — then takes that piece as part of its input to produce the next one, and so on. Each of those steps is called a forward pass: the act of looking at the current text and computing what comes next.

In each step, the model looks at the tokens so far and computes a probability for every possible next token — a likelihood score telling it how likely each candidate word is to be correct — then picks one (usually the most likely). That picked token is added to the sequence, and the whole process repeats. This is sequential: step two cannot start until step one is finished, because the model needs the first token to know what to compute next. There is no way to peek ahead, because the answer is generated, not retrieved.

Multi-token decoding is a set of techniques that try to do the same work in fewer passes. The clever trick is: instead of asking the big, slow model to produce one token at a time, you let a small, fast 'draft' model — a lighter version of an AI designed for speed rather than precision — guess several future tokens in advance. Then you ask the big model to check those guesses all at once.

The big model can verify several guesses in a single forward pass (because checking is parallel — it can score many candidates at once). If the guesses are correct, you are done after one verification, having produced several tokens in one step. If some are wrong, you discard the wrong ones and start over from the last correct guess. The result is exactly the same output you would have gotten the slow way — the big model is the authority, and the draft is just a guess — but you have used fewer sequential passes.

Imagine you are a very thorough, very slow proofreader. Your job is to correct a long manuscript, but you can only look at one word at a time, and after each word you must decide the next word. That is like standard generation. Now imagine you have a quick assistant who is not as accurate but is much faster. The assistant writes down a whole sentence — a guess at what you would write. You then scan the entire sentence at once — because you can check many words in parallel — and you either approve the whole thing or stop at the first mistake. If the assistant is usually right, you save a lot of effort: you correct a whole sentence in one pass instead of word by word. The key is that you are the final authority: if the assistant writes 'the cat sat on the mat' and you agree with every word, you have just produced five tokens in one go. If the assistant writes 'the cat sat on the dog' and you disagree with 'dog', you reject that and everything after it, and you start again from 'on'. The output is identical to what you would have produced on your own, because you are the one who approves every token. The analogy breaks down in one important way: the assistant and the proofreader are not two people — they are two different models.

Multi-token decoding is a method for speeding up AI text generation by having a small, fast model propose several future words at once, which a larger, more accurate model then checks together in a single step, accepting the proposal only up to the point where the two models disagree.

This concept sits inside Large Language Models. Your notes already cover Tokenization and Context Window. Tokenization defines what a token is: the small pieces of text that the model reads and writes. Context Window bounds how many tokens the model can consider at once — relevant because both the guessing and checking steps operate within this limit.