# Speculative Decoding

Laplace’s Mini World · Plate F33 · AI

> One stream waits on reading weights, not arithmetic. A small model guesses; the large one checks, and every word keeps its odds.

- Interactive model: https://world.lapla.si/en/specdec/
- Embed: https://world.lapla.si/embed/en/specdec/
- 中文: https://world.lapla.si/zh/specdec/
- Updated: 2026-10-22

## 01 One Token at a Time

A language model writes one token at a time: the next can only come once the one before it is there. For each token, all of the model’s weights have to be moved from memory to the arithmetic units. Llama 3.1 8B has 8,030,261,248 parameters; in 16-bit numbers that is about 16 GB.

An NVIDIA H100 SXM reads its memory at 3.35 TB a second, so reading 16 GB takes at least about 4.8 milliseconds, and one stream of text gets at most about 209 tokens a second. The arithmetic for that step is about 16 billion operations, which, on paper, the H100 finishes in under 0.02 milliseconds: its arithmetic units spend 99.7% of the time waiting for data. On the bench, the pins on the big drum are the weights, and writing one token takes one full turn of the drum.

*Llama 3.1 8B · about 16 GB · on an H100 at least 4.8 ms a token (on paper) · arithmetic 0.3% busy*

## 02 One Read, Many Streams

Weights read once can serve several streams of text at the same time: put them in a batch, and each weight read is used once for every stream. A batch of 64 still reads the same 16 GB; it does 64 times the arithmetic and, on paper, writes about 13,350 tokens a second, 64 times as many as one stream. On the bench, the reader lights one lane for each stream it works for in a turn.

An H100 can do about 295 operations for each byte it reads; one stream uses about one. But every stream also has its own KV cache to read, and that cannot be shared (the plate Attention in Transformers covers what the cache is): Llama 3.1 8B keeps about 131 KB per token, so 64 streams of 4,096 tokens each add about 34 GB to every step, and the batch writes about 4,250 tokens a second. Pope et al. write that with small batches and short texts the time goes on loading weights, with large batches and long texts on loading the cache. On the bench, the cabinet at the back right has a drawer per stream: whenever the weights are read, each stream’s drawer has to be opened and read too.

*A batch of 64 · weights alone: arithmetic 22% busy · with a 4,096-token cache, about 50 GB read per step*

## 03 Let a Small Model Guess

With one stream, can the idle arithmetic be put to work? Between late 2022 and early 2023, Leviathan and colleagues at Google and Chen and colleagues at DeepMind each gave the same answer. A much smaller model guesses a few tokens ahead, and the large model checks them all at once. Checking a few tokens takes about as long as writing one: it is the same read of the weights, plus arithmetic that was idle anyway. Guessing ahead and checking was older, but only for greedy decoding; what was new was a checking rule that leaves the distribution exactly unchanged when sampling too.

On the bench, the small drum has about a tenth of the big drum’s pins and turns in a tenth of the time. It turns 4 times, and its pencil carriage writes 4 guesses on the tape; then the big drum turns once with 5 lanes lit: it checks the 4 and works out a fifth position. Both models are tiny, counted from the same book (the English page uses Alice’s Adventures in Wonderland): the large one reads the two tokens before, the small one only the one before. Speculative decoding needs nothing from the models but their probabilities for the next token, so the rule and the counts are real; only the models are small, and their sentences make little sense.

*The small model guesses γ = 4 · the large model reads its weights once and works out 5 positions*

## 04 Until the First Wrong Guess

The large model looks at the guesses in order. Each one that passes is kept, and the type head inks it. At the first that fails it stops, puts in a token of its own choosing instead, and throws the rest away, since they follow a wrong token. If all four pass, the pass gives one more for free: the large model has already worked out the fifth position.

So each read of the large model’s weights writes at least 1 token and at most γ + 1. Under every token on the tape is a small mark: a dot for a guess that was kept, a diamond for a token the large model wrote itself (a replacement or the free one).

*Each read of the weights: 1 to γ + 1 tokens · dot = a kept guess · diamond = written by the large model*

## 05 The Rule for Passing

What counts as passing? Say the large model gives the token probability p and the small one gave it q. If q ≤ p, it is kept. If q > p, the small model overrated it, and it is kept only with probability p/q: draw a random number r between 0 and 1, and keep the token if r ≤ p/q. Each gauge on the bridge is one such test: the brass column stands at p/q, full scale is 1, and the red line is r.

When a guess fails, the replacement is not drawn from the large model’s own distribution but from the part where p exceeds q: max(0, p − q), normalised. Tokens the small model under-guessed are made up here; tokens it over-guessed do not come again.

*Kept with probability min(1, p/q) · a replacement comes from max(0, p − q)*

## 06 The Same Distribution

Put together, the chance that any token x is written is min(p, q), for the small model proposing it and its being kept, plus p − min(p, q), for its being put in after a refusal: exactly p. Leviathan and colleagues prove in their appendix that, whatever the small model is, the text has exactly the distribution the large model gives writing one token at a time. Chen and colleagues put it as preserving the target model’s distribution within hardware numerics.

The tubes at the bench’s left end run this experiment. Every trial starts from “, when”; the left tube lets the large model write the next word itself, the right one uses speculative decoding, 50 times each a round. The small model underrates “the” (0.09 against 0.21) and overrates “she” (0.21 against 0.18); in the right tube, brass balls are kept guesses and red ones were put in. Each round scatters by chance; averaged over many rounds, both come to the black marks, the large model’s p (the readout above keeps the running totals). Faster, but neither worse nor better: only the time changes.

*min(p, q) + (p − min(p, q)) = p · the large model’s distribution, unchanged*

## 07 The Acceptance Rate α

The chance that a guess is kept is Σ min(p, q): the more the two distributions overlap, the more often guesses pass. Averaged over the text being written, that is the acceptance rate α. For the pair on the bench α is 0.631. Leviathan and colleagues found α mostly between 0.5 and 0.9 for small models a couple of orders of magnitude smaller than the large one; a bigram model, which reads only the token before, gave α of about 0.2 with T5-XXL on English–German translation.

With γ guesses, each read of the large model’s weights writes on average (1 − α^(γ+1)) / (1 − α) tokens: 2.44 for α = 0.631 and γ = 4; the bench counts 2.46. The formula assumes every position is as easy to guess as every other, and real text is not: after “Mock” comes “Turtle” nearly every time, while the start of a sentence is far harder.

*α = E\[Σ min(p, q)\] · (1 − α^(γ+1)) / (1 − α) tokens per read of the weights*

## 08 How Much Faster

The small model takes time too. If one run of it takes c times as long as one run of the large model, a step takes γc + 1, and the speed goes up by (1 − α^(γ+1)) / ((1 − α)(γc + 1)). Too few guesses waste the chance; too many, and the later ones are mostly thrown away. On the bench, with c = 0.1 and α = 0.631, γ = 3 is best: 1.75 times as fast.

In the papers: T5-XXL (11 billion parameters) with T5-small (77 million), on one TPU v4 and one stream at a time, translated English to German 3.4 times as fast (greedy) and 2.6 times (sampling), with c under 0.05. DeepMind had a 4-billion-parameter model guess for the 70-billion-parameter Chinchilla (1.8 against 14.1 milliseconds a token), with γ = 4: 1.92 to 2.46 times as fast; on two of the tasks it wrote more tokens a second than the bandwidth ceiling for writing one token at a time (model size divided by total memory bandwidth) allows.

*T5-XXL + T5-small · 2.6–3.4× · Chinchilla 70B + 4B · 1.92–2.46×*

## 09 It Spends the Idle Arithmetic

Speculative decoding does not save arithmetic; it does more: thrown-away guesses were computed for nothing, and the small model computes too. Leviathan and colleagues work it out: with α = 0.6 and γ = 2, even leaving the small model’s time and arithmetic out, the large model’s arithmetic goes up 53% and the speed 96%. What it saves is reading: the large model’s weights and KV cache are read once per step, and a step writes several tokens on average. So it pays only while the arithmetic is idle; the authors write that it does not help where no extra computing resources are available.

Batching spends the same idle arithmetic, and the bigger the batch, the less is left for guessing. The EAGLE authors measured 3.01 times on LLaMA2-Chat 70B with one stream, and 2.40 times with a batch of 4; the SmartSpec authors found that past a certain request rate, more guessing makes things slower. With long contexts, though, reading the cache becomes the bottleneck again, and MagicDec still measured up to 2.51 times with batches of 32 to 256. Whether that arithmetic makes one stream fast or many streams cheap is a trade between speed and what each token costs to serve: Pope et al. note that smaller batches give lower latency but a higher cost per token.

*Batching: idle arithmetic for cheapness · speculative decoding: for speed*

## 10 Without a Second Model

A small model has to be found or trained, and served alongside. Later methods go back to an idea of Stern and colleagues (2018) and let the large model guess for itself. Medusa (2024) adds a few small heads to the large model’s final hidden state, the features that go into its output layer: the model’s own output layer guesses the next token, and the k-th head guesses the one k places after that. Several combinations of guesses are checked at once as a tree. EAGLE (2024) starts from the same features, but a very small draft network runs ahead one feature at a time (each step also seeing the token sampled one step earlier), turns them into guesses with the original model’s own output layer, and hands them to the original model to check: 2.7 to 3.5 times as fast on LLaMA2-Chat 70B. Lookahead decoding (2024) needs no draft network: it uses Jacobi iteration to produce several n-grams at once and picks candidates from the n-grams it has stored.

Whether they write what the original model would depends on the checking rule. EAGLE applies the same rule at each branch of a tree of guesses, so the distribution does not change. The lookahead authors also prove that their sampling check leaves it unchanged; lookahead pays in arithmetic instead: in the settings the authors chose on an A100, their 7B, 13B and 34B models do about 120, 80 and 56 times the arithmetic per step, so it only pays when a great deal of arithmetic is idle. With the original model frozen (Medusa-1) and the rule above, the distribution is kept and it still speeds up; but, its authors say, that rule cannot make it any faster, so they offer “typical acceptance” instead. Typical acceptance, and Medusa-2, which fine-tunes the original model with the heads, no longer promise the original model’s distribution. Medusa-2 is 2.3 to 2.8 times as fast.

*Medusa · several heads · EAGLE · running ahead on features · lookahead · Jacobi iteration*

## 11 Inside the Reader

Cut open the reader under the big drum. Each steel tooth rests on a push rod, and the shafts of nine lanes run past all the rods. When a pin goes by, its rod moves, and every lit lane counts it: read once, used several times. The number of lit lanes is the number of tokens worked out at once, streams in a batch or guesses being checked.

The gears in the little mill on the right turn as fast as the arithmetic is busy. With one lane lit they barely move; with all nine, at an H100’s proportions, they use about 3%. It takes about 295 lanes for the arithmetic to keep up with the reading.

*A pin read once, used by every lit lane · nine lanes ≈ 3% of an H100’s arithmetic*

## 12 Taken Apart

A cast-iron bed: at the back left the big drum (the large model) and its reader; at the front right the small drum (the small model) and its pencil carriage; along the front the tape, where the type head writes what the large model accepts and the gauges on the bridge check each guess; in the middle a little mill, the arithmetic; at the back right a cabinet of caches, one drawer per stream; at the left end the tubes that collect results. Each part can be lifted out on its own.

## 13 The Bench

Swap the small model (one that reads the word before, one that reads nothing, one the same as the large model), change how many tokens it guesses, γ, switch to greedy (always the likeliest token), or make the batch bigger and see how the speed changes on paper. The tubes can be emptied and filled again.

**Try this**: Swap the small model for the one that reads nothing: how far does α fall? Then pull γ up to 8: does each turn of the big drum write more, and is the whole faster or slower?

## What the model leaves out

Neither model on the bench is a neural network. Both are counts from the same book: the large one reads the two tokens before, the small one the one before, with Witten–Bell smoothing. On the English page a token is a word or a punctuation mark, on the Chinese page a character; real models use pieces of words (see the plate Tokens and Scaling Laws). Speculative decoding uses nothing from a model but its probabilities for the next token, so the rule, α and the counts on the bench work as in a real system; only the models are small. The drums, pins, lanes and mill are a picture: real weights are numbers in memory, and reading and arithmetic run at the same time on the chip. c = 0.1 is the bench’s setting. The H100 numbers are on paper: reading the weights only (the batch chapter adds the cache), no other overheads; real systems do not reach them.

## A common misconception

It is often thought that speculative decoding swaps the large model for a small one, so the writing gets a little worse. With the rule of Leviathan and of Chen, the text has exactly the distribution of the large model writing on its own: the small model decides how fast, not what may be written or how likely it is, and however badly it guesses, it only makes things slower. On real hardware the order of arithmetic differs and the numbers differ in their last digits, so even greedy decoding can give a different text; that is why Chen and colleagues say the distribution is preserved within hardware numerics. Variants that loosen the rule, such as Medusa’s typical acceptance, can change the distribution.

## Common questions

### Why is text generation by large models slow?

For each token, all of the model’s weights have to be moved from memory to the arithmetic units.

See: [01 One Token at a Time](https://world.lapla.si/en/specdec/#onebyone)

### What is speculative decoding?

A much smaller model guesses a few tokens ahead, and the large model checks them all at once.

See: [03 Let a Small Model Guess](https://world.lapla.si/en/specdec/#draft)

### Does speculative decoding lower the quality of the output?

Leviathan and colleagues prove in their appendix that, whatever the small model is, the text has exactly the distribution the large model gives writing one token at a time.

See: [06 The Same Distribution](https://world.lapla.si/en/specdec/#exact)

### How much faster is speculative decoding?

DeepMind had a 4-billion-parameter model guess for the 70-billion-parameter Chinchilla (1.8 against 14.1 milliseconds a token), with γ = 4: 1.92 to 2.46 times as fast

See: [08 How Much Faster](https://world.lapla.si/en/specdec/#speedup)

## Sources

1. [Leviathan, Kalman, Matias (2023), Fast Inference from Transformers via Speculative Decoding, ICML 2023, arXiv:2211.17192 (the rule and its proof in Appendix A.1, α, Eq. 1, Theorems 3.8 and 3.11, Tables 1–3)](https://arxiv.org/abs/2211.17192)
2. [Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper (2023), Accelerating Large Language Model Decoding with Speculative Sampling, DeepMind, arXiv:2302.01318 (Chinchilla 70B with a 4B draft, 1.8 and 14.1 ms, 1.92–2.46×, “within hardware numerics”)](https://arxiv.org/abs/2302.01318)
3. [Pope et al. (2022), Efficiently Scaling Transformer Inference, Google, arXiv:2211.05102 (weights read once per step, 2N operations per token, when weights or the cache dominate)](https://arxiv.org/abs/2211.05102)
4. [NVIDIA, H100 Tensor Core GPU (H100 SXM: memory bandwidth 3.35 TB/s; BF16 1,979 TFLOPS is with sparsity, dense about half)](https://www.nvidia.com/en-us/data-center/h100/)
5. [Meta, Llama 3.1 8B (8,030,261,248 parameters); Llama Team (2024), The Llama 3 Herd of Models, arXiv:2407.21783, Table 3 (32 layers; model dimension 4,096 over 32 attention heads, so head size 128; 8 key/value heads)](https://huggingface.co/meta-llama/Llama-3.1-8B)
6. [Williams, Waterman, Patterson (2009), Roofline: An Insightful Visual Performance Model for Multicore Architectures, Communications of the ACM 52(4) (the roofline, operational intensity)](https://doi.org/10.1145/1498765.1498785)
7. [Stern, Shazeer, Uszkoreit (2018), Blockwise Parallel Decoding for Deep Autoregressive Models, NeurIPS 2018, arXiv:1811.03115 (the greedy-only forerunner)](https://arxiv.org/abs/1811.03115)
8. [Cai et al. (2024), Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, arXiv:2401.10774 (heads, tree attention, typical acceptance, Medusa-1 and Medusa-2; Medusa-2’s 2.3–2.8× per the paper’s abstract and Table 1, the arXiv listing’s abstract says 2.3–3.6×)](https://arxiv.org/abs/2401.10774)
9. [Li, Wei, Zhang, Zhang (2024), EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, arXiv:2401.15077 (second-to-top-layer features, 2.7–3.5× on LLaMA2-Chat 70B, Table 7 batch and speed)](https://arxiv.org/abs/2401.15077)
10. [Fu, Bailis, Stoica, Zhang (2024), Break the Sequential Dependency of LLM Inference Using Lookahead Decoding, arXiv:2402.02057 (Jacobi iteration, n-grams, sampling check, extra arithmetic)](https://arxiv.org/abs/2402.02057)
11. [Liu et al. (2024), Optimizing Speculative Decoding for Serving Large Language Models Using Goodput (SmartSpec), arXiv:2406.14066 v1 (more guessing gets slower at high request rates)](https://arxiv.org/abs/2406.14066v1)
12. [Sadhukhan et al. (2024), MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding, arXiv:2408.11049 v5 (long contexts, batches of 32 to 256, up to 2.51×)](https://arxiv.org/abs/2408.11049v5)
13. [Lewis Carroll (1865), Alice’s Adventures in Wonderland, Project Gutenberg ebook 11 (the English page’s book, public domain)](https://www.gutenberg.org/ebooks/11)
14. [Lu Xun 鲁迅, 《呐喊》 (Call to Arms), Wikisource (five stories: 故乡, 社戏, 孔乙己, 一件小事, 风波; the Chinese page’s book, public domain)](https://zh.wikisource.org/wiki/%E5%90%B6%E5%96%8A)

## Data

The two models on the bench are counted by the page from one book each: Lewis Carroll, Alice’s Adventures in Wonderland (1865; Project Gutenberg ebook 11, public domain) for English, and five stories from 鲁迅《呐喊》 (1919–1922; Wikisource, public domain) for Chinese (src/topics/specdec/corpus.py, sim.js). Nothing on the bench is the output of a real language model. Material presets, parts and the softbox studio of the site’s model kit are ported from AirsupHQ/machine-skill (MIT, © 2026 Air Supply Technologies Inc.).

## Licence CC BY-NC 4.0

The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.

https://creativecommons.org/licenses/by-nc/4.0/deed.en

## Cite this plate

Laplace’s Mini World, Plate F33, “Speculative Decoding”, https://world.lapla.si/en/specdec/, CC BY-NC 4.0, updated 2026-10-22.