AI glossary
Speculative Decoding
Speculative decoding is an inference-time technique that accelerates text generation from a large language model by having a smaller, faster draft model propose several candidate tokens ahead of time, which the larger target model then verifies in a single pass, without changing the final output.
Standard autoregressive generation produces one token at a time. The model computes a full forward pass to predict the next token, then repeats that entire process for every subsequent token. This step-by-step approach is slow for large models because each individual forward pass is expensive. It also under-utilizes modern hardware like GPUs, which are much more efficient at processing many tokens in parallel, as happens during training, than at generating one token sequentially.
How speculative decoding works
The process begins with a smaller, faster draft model. This model generates a short sequence of candidate next tokens—often several tokens ahead—at a much lower computational cost than the large model could.
Next, the larger target model processes all of these candidate tokens together in a single forward pass. This is computationally similar in cost to processing just one token because GPUs can parallelize across the sequence.
The target model then checks the candidates token by token, starting from the beginning of the sequence. It accepts every candidate token that matches what it would have generated itself. At the first point where the target model disagrees with the draft, it discards the rest of the draft’s candidates. It then generates the correct next token itself by resampling from that position. The cycle repeats with a new draft.
Why it is needed
The primary driver for this approach is hardware efficiency. Standard inference of large language models is bottlenecked by the sequential nature of token prediction. Each token requires a full forward pass through the transformer architecture.
Modern GPUs excel at parallel computation. By grouping multiple token predictions into a single forward pass, speculative decoding aligns better with how these chips operate. Instead of waiting for one slow pass to complete before starting the next, the system processes a block of tokens simultaneously. This reduces the total number of forward passes required to generate a response, significantly cutting down latency.
Why it does not change the output
A common concern is whether using a draft model alters the quality or randomness of the generated text. It does not. The target model still makes every final decision about which tokens are accepted. It uses the exact same probability distribution it would use for standard generation.
The draft model only proposes candidates that get checked. They are not accepted automatically. Acceptance is decided using the target model’s own output distribution rather than the draft model’s. Because of this, the technique is designed so that the final generated text has the same statistical properties as if the target model had generated it token by token on its own. It changes only the speed, not the sampling behavior.
Origin
The technique was introduced in two papers published in 2023. “Fast Inference from Transformers via Speculative Decoding” was authored by Yaniv Leviathan, Matan Kalman, and Yossi Matias from Google Research. Independently, Charlie Chen and colleagues at DeepMind published “Accelerating Large Language Model Decoding with Speculative Sampling.”
Both works described the same core idea: using a smaller, faster draft model to propose tokens that a larger target model verifies in parallel. This convergence highlighted the utility of the approach across different architectures and inference engines.
Limitations
Speculative decoding is not a universal fix. It requires a smaller draft model that is well-aligned with the target model’s behavior. If the draft model frequently disagrees with the target, most proposed tokens get rejected, providing little or no speedup. The achievable speedup depends heavily on how often the draft model’s guesses are accepted, which varies by task and by how well-matched the draft and target models are.
There is also an engineering cost. Running both a draft model and a target model adds complexity and additional memory overhead compared to running the target model alone. You must manage the transfer of tokens between models and coordinate the parallel processing on the GPU. The overhead must be justified by the reduction in total forward passes.
FAQ
How does speculative decoding work in practice?
A small draft model predicts a sequence of tokens. A large target model verifies them in a single parallel pass, accepting matches and rejecting mismatches, thereby reducing the total number of expensive forward passes.
Does speculative decoding change the output of an LLM?
No. The target model uses its own probability distribution to accept or reject tokens. The final text maintains the same statistical properties as standard autoregressive generation.
What is the role of the draft model?
The draft model proposes candidate tokens quickly and cheaply. It acts as a predictor that allows the target model to process multiple tokens in parallel, though it must be well-aligned with the target to be effective.
Why is GPU parallelization important for this technique?
GPUs are optimized for parallel computation. Speculative decoding allows the model to process a block of tokens simultaneously rather than sequentially, making better use of the hardware’s throughput during inference.