# Diffusion Language Models Laplace’s Mini World · Plate F64 · AI > Left to right, each token takes a pass. A diffusion language model lays out blanks, guesses them all each round and keeps the surest; fewer rounds are faster, and more often wrong. - Interactive model: https://world.lapla.si/en/diffusionlm/ - Embed: https://world.lapla.si/embed/en/diffusionlm/ - 中文: https://world.lapla.si/zh/diffusionlm/ - Updated: 2026-11-05 ## 01 Two Ways to Write Most chat models write left to right: each forward pass writes the next token. In 2025 several language models wrote another way: lay out the whole answer as blanks, then fill them round by round, many at a time. They are called diffusion language models; LLaDA 8B, Inception Labs’ Mercury and Google DeepMind’s Gemini Diffusion are three. On the stone, two ferry handbills are set in wood type, every letter chosen by this page’s one toy model. The brass galley behind is set in order, one sort after another, and its counter adds one for each; the iron chase in front is filled the diffusion way, a batch each round, and its dial counts the rounds. Each has 72 sorts: 72 passes behind, 8 rounds in front. *72 sorts each · left to right: 72 forward passes · diffusion: 8 rounds* ## 02 In Order, One After Another A left-⁠to-⁠right language model is called autoregressive. Each forward pass reads every token written so far, works out the odds of the next one, picks one and writes it, then goes on to the next. Writing n tokens takes n passes, one after another: the next waits for the last. The page in the brass galley is set that way: one sort set, the counter adds one, then the next. Type faces are cut in reverse, so seen from the front each line runs right to left and the first line is the far one; on paper they print the right way round. Another way round one token at a time is to let a small model guess a few and the large one check them in one pass; see “Speculative Decoding”. *72 sorts · 72 forward passes · one each* ## 03 A Page of Blanks A diffusion language model first fixes how long the answer will be and puts a special mask token in every place, meaning: a token goes here, not yet decided. LLaDA treats the answer’s length as a setting chosen beforehand. Compositors had an old trick: short of a letter, they set another of the same width upside down to hold its place, foot up, and swapped in the right letter later. The blanks in the chase in front are such turned sorts: 72 places, all blank, the dial at 0. Picture diffusion starts from noise and removes it step by step; here the “noise” is the mask: adding noise turns tokens into masks, removing it fills masks back with tokens. For the kind that makes pictures and video, see “Video Diffusion”. *72 blanks · round 0 · each blank is a mask token* ## 04 A Guess for Every Blank Each round the model reads the whole page, the set type and every blank, seeing both left and right, because its Transformer has no causal mask. In one forward pass it gives, for every blank, the odds of every token; the likeliest is its guess, and those odds are how sure it is. On the stone, a wooden sort rises over every blank: this round’s guesses. The “model” here is this page’s toy: it has read only a few handbills like this one and counts how often each letter stood in each place. A real diffusion language model learns the same thing with a neural network, from trillions of tokens. *1 forward pass · a guess for each of the 72 blanks · each with its odds* ## 05 Keep the Surest After guessing, only some are kept. By default LLaDA keeps the surest and turns the rest back into masks: “low-⁠confidence remasking”. How many each round is fixed beforehand: 72 in 8 rounds, 9 a round. The places turned back are guessed again next round, when more of the page is set around them. In this method a token once set is not changed again. The ones to keep can also be picked at random. In the LLaDA paper, with answers 256 tokens long and 256 steps, keeping the surest scored 70.0 on the GSM8K maths problems; keeping at random, 21.3. *8 rounds · 9 kept a round · the rest masked again · GSM8K: surest 70.0, random 21.3 (256 steps)* ## 06 Taught to Fill Blanks Training is a fill-⁠in-⁠the-⁠blanks exercise: take real text, draw a share t between 0 and 1, turn each token into a mask with probability t, and have the model guess the hidden tokens, counting its errors only where they are hidden. A small t is a page nearly done; t near 1, a page nearly blank. So every stage, from all blank to one letter short, has been practised. LLaDA’s loss is also multiplied by 1/t; the paper states that this loss is an upper bound on the negative log-⁠likelihood, so the model is a generative model. BERT fills blanks too, but at one fixed share. LLaDA 8B was trained from scratch on 2.3 trillion tokens, with 0.13 million H800 GPU hours, then fine-⁠tuned on 4.5 million pairs. *A random share t of tokens masked · loss only where masked · on the stone t = 15%, 50%, 85%* ## 07 Fewer Rounds, Crossed Words In 8 rounds, 72 sorts go in 9 at a time; in 1 round, all 72 at once. Tokens set in the same round cannot see each other: each blank takes its own likeliest token, and together they may not fit. A paper on speeding these models up gives an example: a two-⁠word poker hand might be “high card” or “full house”; guessed separately, the words can come out as “high house”. The page on the stone is set in 1 round, and 1 line comes out as one it has never read (in red in the HUD). LLaDA’s authors report the best results when the steps equal the answer’s length, one token a step; the scores in the paper’s tables were made that way. Fewer rounds are faster, but score lower. *1 round: 72 at once, 1 line garbled · 8 rounds: none · LLaDA’s table scores: one token a step* ## 08 Where the Speed Comes From A diffusion language model saves passes: 72 sorts, 8 forward passes. But each pass reads the whole page again, prompt included. A left-⁠to-⁠right model keeps a KV cache and need not recompute what it has done; this kind of diffusion model does not. Fewer passes are not less arithmetic: for this page, diffusion works through 8 × 72 = 576 positions, left to right only 72. Speeds the makers report: Inception Labs says Mercury Coder Mini writes about 1,⁠109 tokens a second on NVIDIA H100s (February 2025, API speed measured by Artificial Analysis; in the same test Gemini 2.0 Flash Lite wrote 201). Google DeepMind says Gemini Diffusion samples 1,⁠479 tokens a second, not counting 0.84 seconds of overhead; it is still an experimental demo. Each maker measured on its own chips and its own tasks, so the numbers cannot be compared directly. Open diffusion language models are often slower in practice than autoregressive ones. Fast-⁠dLLM gave LLaDA an approximate cache and sets several tokens at once only when sure enough: about 8 times faster on an A100, with GSM8K going from 79.3 to 78.5. *This page: diffusion 8 passes, 576 positions · left to right 72 and 72 · makers’ speeds: Mercury Coder Mini about 1,⁠109 tokens/s (H100), Gemini Diffusion 1,⁠479 tokens/s (overhead not counted)* ## 09 Both Sides in View A left-⁠to-⁠right model, partway through, sees only what is to its left. A diffusion model sees the whole page every round, so a gap in the middle can be filled with both sides in view. On the stone, the middle of the second and third lines is blank and both ends are set; the model fills the gap from both ends: “TO … RTH” leaves only NORTH. The LLaDA paper also tested poems: on the authors’ own set of 496 pairs of lines from Chinese poems, asked for the line before a given one, LLaDA 8B scored 45.6 and GPT-⁠4o 34.3; asked for the line after, LLaDA scored 51.8 and GPT-⁠4o 82.7. That is one paper, on one set of poems. *Both ends given, the middle filled · 4 rounds · the line before: LLaDA 45.6, GPT-⁠4o 34.3 · the line after: 51.8, 82.7* ## 10 Where It Falls Short The answer’s length must be fixed first; LLaDA treats it as a setting and fills the places left over with end tokens. Few rounds garble words, many rounds are slow, and there is no ready-⁠made KV cache. On scores, LLaDA 8B’s base model and LLaMA3 8B of the same size each win some: MMLU 65.9 to 65.4 and GSM8K 70.3 to 48.7, but PIQA 73.6 to 80.6 and MBPP 40.0 to 48.8. In Google DeepMind’s own table, Gemini Diffusion codes about as well as Gemini 2.0 Flash-⁠Lite but trails it on GPQA Diamond, 40.4% to 56.5%, and Global MMLU, 69.1% to 79.0%. Masked diffusion is one kind of diffusion language model: others denoise continuous word vectors, turn tokens into random tokens, or train with “score entropy”. All of them still predict tokens from probabilities. *Length fixed first · few rounds garble · no ready-⁠made KV cache · scores win some, lose some* ## 11 Taken Apart On the stone: the iron chase with its furniture and quoins locks the page in front, and the dial counts rounds; behind it, the page set in order in a brass galley, and a counter for the passes. The type is maple wood type, its faces cut in reverse; the blanks are turned sorts, foot up. *Chase · furniture and quoins · the diffusion page · round dial · galley · the left-⁠to-⁠right page · counter* ## 12 The Bench Pick a number of rounds and watch the page fill: 1 round sets all 72 at once, 72 rounds one a round. “Keep at random” keeps a random choice of each round’s guesses instead of the surest; “Set it again” draws a new random order. The model here is this page’s toy: for each line it has read only three versions. The readings count forward passes only; they are not running times. *A toy model · counts forward passes only · not running times* **Try this**: In the lab, pick 1 round: with all 72 sorts set at once, how many lines come out as ones the model never read? And with 8 rounds? ## What the model leaves out The wood type, the chase, the galley, the dial and the counter are this page’s model. The handbill has 6 lines; for each line the “model” has read only three versions and takes how often it read them as its odds: a toy, not a real language model, and it cannot see the other lines. One letter stands for one token; a real model’s tokens are words or pieces of words. The guesses rise and fall one by one so they can be seen; a real model gives the odds for every blank in one forward pass. Every sort here is one width; real type varies. ## Easy to get wrong A diffusion language model does not “think in parallel” or edit like a person: each round it gives every blank a token from the odds, then keeps some by a rule; in LLaDA’s method a token once set is not changed. “Faster” depends on the rounds, the answer’s length, the hardware and whether there is a cache; makers’ speeds are their own, on their own chips, and cannot be compared directly. The scores in LLaDA’s tables were made one token a step. The poem result comes from one paper and one set of Chinese poems. Masked diffusion is one kind of diffusion language model among several. The compositor’s turned sort is only a likeness: the sort that replaces it is the right letter, while the model fills in what it finds likeliest. ## Common questions ### What is a diffusion language model? Most chat models write left to right: each forward pass writes the next token. In 2025 several language models wrote another way: lay out the whole answer as blanks, then fill them round by round, many at a time. See: [01 Two Ways to Write](https://world.lapla.si/en/diffusionlm/#whole) ### How does an autoregressive language model write? A left-⁠to-⁠right language model is called autoregressive. Each forward pass reads every token written so far, works out the odds of the next one, picks one and writes it, then goes on to the next. See: [02 In Order, One After Another](https://world.lapla.si/en/diffusionlm/#inorder) ### Which tokens does a diffusion language model keep each round? After guessing, only some are kept. By default LLaDA keeps the surest and turns the rest back into masks: “low-⁠confidence remasking”. How many each round is fixed beforehand: 72 in 8 rounds, 9 a round. See: [05 Keep the Surest](https://world.lapla.si/en/diffusionlm/#keep) ### How is a diffusion language model trained? Training is a fill-⁠in-⁠the-⁠blanks exercise: take real text, draw a share t between 0 and 1, turn each token into a mask with probability t, and have the model guess the hidden tokens, counting its errors only where they are hidden. See: [06 Taught to Fill Blanks](https://world.lapla.si/en/diffusionlm/#learn) ### Is a diffusion language model always faster? A diffusion language model saves passes: 72 sorts, 8 forward passes. But each pass reads the whole page again, prompt included. A left-⁠to-⁠right model keeps a KV cache and need not recompute what it has done; this kind of diffusion model does not. See: [08 Where the Speed Comes From](https://world.lapla.si/en/diffusionlm/#speed) ## Sources 1. [S. Nie et al. (2025): Large Language Diffusion Models, arXiv:2502.09992 (LLaDA: an 8B masked diffusion language model; training, sampling, remasking; the reversal poems)](https://arxiv.org/abs/2502.09992) 2. [J. Austin et al. (2021): Structured Denoising Diffusion Models in Discrete State-⁠Spaces, NeurIPS 2021 (D3PM: noise that turns tokens into a mask, the absorbing state)](https://arxiv.org/abs/2107.03006) 3. [S. S. Sahoo et al. (2024): Simple and Effective Masked Diffusion Language Models, NeurIPS 2024 (MDLM: the objective is a weighted sum of fill-⁠in losses at every masking ratio)](https://arxiv.org/abs/2406.07524) 4. [Inception Labs (2025): Mercury: Ultra-⁠Fast Language Models Based on Diffusion, arXiv:2506.17298 (speeds as the maker reports them)](https://arxiv.org/abs/2506.17298) 5. [Google DeepMind: Gemini Diffusion (the page of an experimental model; speed and benchmarks as the maker reports them)](https://deepmind.google/models/gemini-diffusion/) 6. [C. Wu et al. (2025): Fast-⁠dLLM, arXiv:2505.22618 (diffusion language models lack a KV cache; setting too many tokens at once hurts quality)](https://arxiv.org/abs/2505.22618) ## Data Masked diffusion for language after S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen and C. Li, “Large Language Diffusion Models” (LLaDA; arXiv:2502.09992); J. Austin, D. D. Johnson, J. Ho, D. Tarlow and R. van den Berg, “Structured Denoising Diffusion Models in Discrete State-⁠Spaces” (D3PM; NeurIPS 2021); S. S. Sahoo et al., “Simple and Effective Masked Diffusion Language Models” (MDLM; NeurIPS 2024). Speeds as reported by Inception Labs (Mercury, arXiv:2506.17298) and Google DeepMind (Gemini Diffusion); faster sampling and the cost of decoding many tokens at once after C. Wu et al., “Fast-⁠dLLM” (arXiv:2505.22618). The handbill, its wood type, the chase, the galley and the counters are this page’s model, not a laboratory’s; the toy model that fills them knows only the lines listed in docs/plates/diffusionlm/computed.py. Material presets and the model kit as on the site’s other model plates (public/looks/CREDITS.md). ## Licence CC BY-NC 4.0 The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence. https://creativecommons.org/licenses/by-nc/4.0/deed.en ## Cite this plate Laplace’s Mini World, Plate F64, “Diffusion Language Models”, https://world.lapla.si/en/diffusionlm/, CC BY-NC 4.0, updated 2026-11-05.