Video Diffusion

What they make now

In February 2024 OpenAI published a technical report on Sora: it makes up to a minute of high-definition video, widescreen 1920 × 1080, vertical 1080 × 1920 or anything between. The report gives the method (compress the video, cut it into “spacetime patches”, hand them to a diffusion transformer) and says that model and implementation details are not included.

Open models put the details in their papers. Tencent’s HunyuanVideo (December 2024) has over 13 billion parameters and by default makes 129 frames at 720 × 1280. Alibaba’s Wan2.1 (2025) comes in 14 billion and 1.3 billion parameters and by default makes 81 frames at 16 a second, about 5 seconds. Veo 3 (2025) makes sound and picture together; in the Gemini API in July 2025 a clip was 8 seconds, 720p, 24 frames a second.

The plinth is a small version of what the open models share, slow enough to see every step: a clip of 17 frames, a red ball bouncing on a wooden table.

Sora: up to a minute (2024) · HunyuanVideo: 13 billion parameters · Wan2.1: 14 billion / 1.3 billion · Veo 3: 8 seconds with sound (Gemini API, July 2025)

A clip is a block

Video is pictures in a row in time. This clip has 16 frames a second and 17 frames, frame 0 to frame 16: exactly one second. Each glass slide is a frame, and time runs left to right along the rail.

Stack the frames and you get a block with a width, a height and a length in time. The ball’s place in each frame, joined up, draws an arc through the glass: its path in space and its path in time at once.

A video model works on the whole block. Sora’s report treats pictures the same way: “images are just videos with a single frame”.

17 frames · 16 per second · 1 second · time runs left to right

Compress first

Working on the pixels is too expensive, so a video autoencoder (a VAE) first squeezes the block. HunyuanVideo’s and Wan2.1’s both shrink time 4 times and height and width 8 times each, and keep 16 numbers at each place, the 16 channels. Everything after happens in this compressed “latent space”.

It is causal: frame 0 becomes a latent frame of its own, then every 4 frames become one, and later frames never change earlier ones. Wan’s paper says the first frame is compressed only in space, so pictures and videos share one VAE. The 17 frames become 5 latent frames: the slides give way to five laminated slabs, and the 16 layers on their edges stand for the 16 channels.

The clip on the plinth is 17 × 144 × 256 pixels with 3 numbers each, 1,880,064 numbers; the latent is 5 × 18 × 32 × 16 = 46,080, about 41 times fewer. For a 720p, 81-frame clip from Wan2.1 it is about 46 times fewer.

time 4× · height and width 8× each · 16 channels · frame 0 alone · 17 frames → 5 latent frames

Cut into spacetime patches

The latent is cut again: 1 latent frame in time, 2 cells in height and 2 in width. HunyuanVideo and Wan2.1 both cut it this way. Each piece is one “token” for the transformer, as a word piece is for a language model; Sora’s report calls them spacetime patches.

Here 5 latent frames of 9 × 16 pieces make 720 tokens. The one lifted out covers 4 frames in time and 16 × 16 pixels of the picture; it holds 2 × 2 cells of 16 numbers, 64 numbers. A linear layer turns it into a longer row of numbers: 1,536 in Wan2.1’s 1.3-billion model, 5,120 in its 14-billion one.

How many tokens a clip makes depends on its size and length. The DiT paper cuts an image into patches of side p; halving p makes 4 times the tokens. Sora’s report says the size of the grid of random patches set out at the start decides the size of the video.

patches of 1 × 2 × 2 · 5 × 9 × 16 = 720 tokens · 64 numbers each

Adding noise

In training, noise is added to the latent of a real clip and the model learns to take it away. Models like this are called diffusion models. Wan2.1, HunyuanVideo, Movie Gen and Open-Sora 2.0 are trained with a variant called flow matching: a straight line between the clip and the noise, x_t = (1 − t)·x₀ + t·ε. At t = 1 it is all noise; at t = 0 it is the clip.

Wherever the dial points, the glass shows that point on the line: the picture fades out and the grain fades in. The noise goes on the latent, an independent Gaussian number in every cell of every latent frame.

The older diffusion model DDPM (2020) adds noise in 1,000 steps along a path that is not straight, and its network learns to guess the noise that was added. The straight path comes from two papers of 2022, rectified flow and flow matching; Stable Diffusion 3 used it in large transformers. The page writes t as Stable Diffusion 3 does, t = 1 for noise; Wan’s paper writes it the other way round.

x_t = (1 − t)·x₀ + t·ε · t = 1 all noise · t = 0 the clip · DDPM: 1,000 steps

Removing it, step by step

Making a clip runs the other way. It starts from a latent of pure noise; at each step the transformer looks at the whole block, estimates which way to go (in flow matching, the velocity) and takes a small step. Wan2.1 and HunyuanVideo take 50 steps by default. Every frame clears in the same steps; frames are not drawn one after another.

The denoiser on the plinth is a small stand-in. It knows only 8 clips of a bouncing ball, and at each step it works out the best guess: those 8 clips averaged by weight. The bars at the top right are the 8 weights. At first all 8 are equally likely and the glass holds a haze of faint balls; the weight gathers as it goes, and at the end one clip is left. Other starting noise can end on another clip.

A real model is a trained network, not a formula, and it does more than copy clips it has seen. At each step Wan2.1 runs its network twice, with the prompt and without, and combines the two with a guidance scale of 5.0, so 50 steps take 100 passes.

50 steps by default (Wan2.1, HunyuanVideo) · every frame at once · here: the exact denoiser for 8 clips

Looking across space and time

In a transformer every token looks at the others and takes in what it needs from each: that is attention (how it is computed is the plate Transformer attention). In video, a token can look elsewhere in the same moment, or at other moments.

Earlier video diffusion models split the two: first within a frame, then along time at one place, to save computation. Video Diffusion Models (2022) and Latte (2024) work this way. CogVideoX, Movie Gen, HunyuanVideo and Wan2.1 use full attention instead: every token looks straight at every token, at every moment and every place. The CogVideoX paper says the split version struggles to keep objects that move a lot consistent.

The threads start at the chosen token, which holds the ball, and run to the tokens it looks at most: mostly the pieces that hold the ball at other moments. The page computes this from a toy score of “similar content”; it is not any model’s learned attention. The bench can limit it to the same moment, or to the same place.

full attention: CogVideoX · Movie Gen · HunyuanVideo · Wan2.1 · split in space and time: VDM (2022) · Latte (2024)

How the prompt gets in

A language model first turns the prompt into a row of vectors. Wan2.1 uses umT5, which reads Chinese and English, up to 512 tokens, and adds a cross-attention layer to every transformer block, where the video tokens look at the text tokens. HunyuanVideo uses a multimodal large language model and joins the text and video tokens into one sequence that shares the same attention.

The network also needs to know how far the denoising has got. DiT uses that to set each layer’s scale and shift (adaLN); HunyuanVideo adds a vector for the whole sentence from CLIP. In training the prompt is sometimes dropped, and in sampling each step runs twice, with and without the prompt, and the difference is pushed further: classifier-free guidance. Sora’s report says its training videos were recaptioned by a model that writes detailed descriptions, and GPT expands a user’s short prompt into a long one.

On the plinth, the plate’s three groups of words run to their tokens: “a red ball” to the pieces that hold the ball, “a wooden table” to the bottom row, “bounces” to the ball at each moment. This is a sketch. Changing the prompt on the bench makes the denoiser choose only among that prompt’s 8 clips.

Wan2.1: umT5 + cross-attention · HunyuanVideo: multimodal LLM, text and video in one sequence · classifier-free guidance

Back to pixels

When the denoising ends, the result is still a latent. The VAE’s decoder turns the 5 latent frames back into 17 frames of picture; it runs once at the end, not at every step.

Decoding has to put back the detail the compression left out, and that is also its limit: however good, the reconstruction is not the original. A long clip does not fit in memory at once: HunyuanVideo splits it into overlapping tiles, encodes and decodes each and stitches them; Wan’s VAE takes 4 frames at a time and caches what came before, which its paper says lets it handle videos of any length.

On the plinth the five slabs open back into 17 slides, and the coarse cells sharpen into the ball.

5 latent frames → 17 frames · decoded once

The cost

Tokens grow with frames and with pixels. An 81-frame clip from Wan2.1 is 32,760 tokens at 480p and 75,600 at 720p; HunyuanVideo’s 129 frames at 720p are 118,800. The clip on the plinth is 720.

Full attention compares every token with every token, so the pairs grow as the square: about 5.7 × 10⁹ per head per layer for Wan2.1 at 720p, about 1.4 × 10¹⁰ for HunyuanVideo. Wan’s paper says the cost of attention grows with the square of the tokens, and at a million tokens attention takes up to 95 % of the training time.

So it is slow. Wan2.1’s 14-billion model takes 1,837.9 seconds, about 31 minutes, for one 720p clip on one H800 / H100. The 1.3-billion model takes about 4.4 minutes for 480p on one RTX 4090, in 8.19 GB of memory (with parts of the model and the text encoder moved off the GPU).

tokens ∝ frames × pixels · attention ∝ tokens² · Wan2.1 720p: 75,600 tokens, about 31 minutes (one H800 / H100)

Where physics breaks

Sora’s own report says it does not model the physics of many basic interactions accurately, glass shattering for one; eating food does not always change the food; long clips drift out of step and objects appear from nowhere. Sora 2 (2025) says it is better at obeying the laws of physics than earlier systems, and also that it is far from perfect.

Physics-IQ (2025) asks models to continue real filmed physics, scoring the difference between two real takes as 100. Among the 2024-era models tested the best scored 29.5, and Sora (from a single image) 10.0; the authors conclude that visual realism does not imply physical understanding. VideoPhy (2024) tested 12 models: the best, CogVideoX-5B, made videos that matched the prompt and physical common sense 39.6 % of the time; every other model was under 20 %.

Kang and colleagues (2024) trained video models of up to 310 million parameters from scratch on 2-D videos of balls. They were near perfect inside the range they were trained on and wrong outside it. The models seemed to copy the closest training example rather than learn the law, and leaned on color first, then size, then velocity, then shape.

The glass shows two recorded kinds of error: at its first bounce the ball sinks into the table (Physics-IQ notes objects passing through other objects), and from frame 10 a second ball appears from nowhere (the “spontaneous appearances” of Sora’s report). The page drew them; they are no model’s output.

Physics-IQ: best 29.5 of 100 · VideoPhy: best 39.6 % · Kang et al.: color > size > velocity > shape

Taken apart

The cassette in its parts: 17 glass slides, the rail, two aluminium frames, two combs, the noise dial and the prompt plate. Whichever the pointer rests on is lifted out and named.