# Vision-Language-Action Models

Laplace’s Mini World · Plate F44 · AI

> An image and a sentence go in; a string of arm motions comes out.

- Interactive model: https://world.lapla.si/en/vla/
- Embed: https://world.lapla.si/embed/en/vla/
- 中文: https://world.lapla.si/zh/vla/
- Updated: 2026-10-24

## 01 What they do now

In 2023 Google DeepMind’s RT-2 turned a large model trained on web images and text into one that drives a robot: a camera image and an instruction go in, arm actions come out. With objects, backgrounds and rooms it had not seen, its average success rate was 62 %; its predecessor RT-1 managed 32 %.

OpenVLA (2024) has 7 billion parameters, was trained on 970,000 robot demonstrations, and its weights are public. The same year Physical Intelligence’s π0 folded laundry, cleared tables and built boxes, tasks that take 5 to 20 minutes each. In 2025 π0.5 tidied kitchens and bedrooms in homes it had never been in, and Google DeepMind’s Gemini Robotics, after specialised training, averaged 79 % on six dexterous tasks, folding an origami fox among them; packing a lunch box, a job of over two minutes, worked in 20 tries out of 20.

On the bench is a model of a robot cell: an arm with five joints, a depth camera on a mast, a cup, a cloth and a wooden block. The controller on the right stands for the model itself: the amber display shows the instruction, the seven scales on its sloped face the action it has just written. The arm’s path is planned by the page and written in RT-2’s format; it is not a trained model’s output.

*RT-2: unseen scenes 62 % (RT-1 32 %) · OpenVLA: 7 billion parameters, 970,000 demonstrations · π0: laundry, tables · Gemini Robotics: lunch boxes 20 / 20*

## 02 Pixels in

First the model looks. The camera image is cut into small patches; a vision encoder turns each one into a list of numbers, an image token. OpenVLA sends each patch through two encoders, DINOv2 and SigLIP, and joins their outputs; the two together have about 600 million parameters.

The images are small. OpenVLA uses 224 × 224 pixels; it tried 384 × 384, saw no difference in results, and trained 3 times slower. RT-1 looks at its last 6 frames, 300 × 300 pixels each; π0 takes 2 or 3 cameras at a time.

On the bench, the depth camera on the mast watches the work area; top right is the view from its position, cut into patches (their size is only a picture). The gripper carries a small camera too: π0.5’s arms have one at the wrist.

*Image → patches → vision encoder → image tokens · OpenVLA: 224 × 224 pixels, DINOv2 + SigLIP*

## 03 Words in

The instruction is cut into tokens too, placed after the image tokens, and both go into a large language model. That model was first trained on images and text from the web: RT-2 uses PaLI-X and PaLM-E, OpenVLA uses Llama 2, π0 uses PaliGemma. RT-2 asks the way a picture quiz does: “Q: what action should the robot take to \[instruction\]? A:”

Why start from such a model? The web holds billions of images and captions; robot data comes nowhere near: even the largest robot datasets have 100,000 to a million demonstrations. RT-2 kept training on robot data mixed with the web data; with PaLI-X, robot data made up about half. With the same 5 billion parameters, trained from scratch on robot data alone, it succeeded 9 % of the time in unseen scenes; trained on the mix, 44 %.

What it gains is common sense. RT-2 follows instructions that are not in its robot data, such as “move apple to 3”, “move X near the sum of two plus one” or “move the coke can to the person with glasses”. It does not learn new motions: it can only make the moves that appear in the robot data, used in new places.

*Image tokens + instruction tokens → language model · RT-2 at 5 billion parameters: from scratch 9 %, mixed training 44 %*

## 04 Actions as tokens

How do actions come out of a language model? RT-2 writes them as text. An action has 7 continuous quantities: how far the hand moves in x, y and z, how far it turns about three axes, and how wide the gripper opens. Each one’s range is cut into 256 equal bins, and the bin number is written down; with one token for “stop” in front, an action is 8 whole numbers, such as “1 ​128 ​91 ​241 ​5 ​101 ​127 ​217”.

These 256 numbers need 256 tokens of the vocabulary. PaLI-X already has a token for every whole number up to 1,000 and uses those; PaLM-E and OpenVLA take over the 256 least used tokens. OpenVLA cuts not the full range of each quantity but the 1st to 99th percentile of the training data, so that a few extreme actions don’t stretch the bins.

The 7 scales on the sloped panel are these 7 quantities, 256 bins each. With every step of the arm the riders jump to that step’s bins, and the tape prints the same 7 numbers. A bin is 1/256 of the range, so rounding is off by at most half a bin. Here 5 actions are written a second; RT-2’s 55-billion-parameter version manages 1 to 3.

*7 quantities × 256 bins + stop = 8 tokens · e.g. 1 ​128 ​91 ​241 ​5 ​101 ​127 ​217*

## 05 Look, then move

The robot works in a loop: take a picture, the model writes an action, the arm carries it out, take the next picture. The action says how far the hand should move; a low-level controller turns that into joint angles and follows them. π0.5 has simple PD controllers track the target poses the model gives.

How fast is the loop? RT-1, with 35 million parameters, runs 3 steps a second and leaves the model under 100 ms. RT-2’s 55-billion-parameter version runs on TPUs in the cloud, called over the network, at 1 to 3 steps a second; the 5-billion version at about 5. OpenVLA runs about 6 a second on one RTX 4090.

A person takes 2 to 4 seconds for such a move, so 3 steps a second keeps up. Dexterous work like folding clothes needs more: the ALOHA two-arm rig sends 50 commands a second. In one study, the same teleoperated task took people 62 % longer when commands dropped from 50 a second to 5. The arm on the bench now moves a step, then stops to wait for the next action.

*Picture → model → action → joints · RT-1: 3 steps a second · RT-2 55B: 1⁠–⁠3 · OpenVLA: about 6*

## 06 A chunk at a time

The way out is to write a string of actions at once. ACT (2023) predicts the next 100 actions each time, played at 50 a second, so 2 seconds; in its experiments, 1 action at a time succeeded 1 % of the time, 100 at a time 44 %. Writing a string at once cuts the task into fewer pieces, so small errors pile up less.

π0 writes 50 actions at a time, 1 second at 50 a second. One pass takes 73 ms on an RTX 4090, while the arm plays only 3.65 actions. In practice it looks at a new image every 25 actions, every 0.5 seconds, and writes a new string. Gemini Robotics’ large model sits in the cloud, and a string of actions arrives about 250 ms after the picture; thanks to the strings, the arm still runs at 50 actions a second.

So “50 Hz” is the beat of the actions, not of the model. The beads ahead of the gripper on the bench are the string the model has just written; the arm follows them, and halfway along, a new string joins on.

*π0: strings of 50 actions, 50 a second · one pass 73 ms · a new string every 0.5 s · Gemini Robotics: about 250 ms*

## 07 Actions out of noise

If every quantity of every step of a smooth motion is one token, there are too many: 14 quantities at 50 steps a second make 700 tokens in 1 second. Neighbouring tokens are nearly the same, and a model easily learns to copy the last one.

π0 writes its output differently. Beside the language model sits a 300-million-parameter “action expert” that makes smooth actions directly with flow matching, a kind of diffusion. It starts from a string of random noise; at each step the model says which way to push; after 10 steps the noise is a smooth string of actions. The 10 steps take 27 ms on an RTX 4090. π0 has 3.3 billion parameters in all.

Another way is to compress the actions, then write them as tokens. FAST applies the discrete cosine transform to each quantity (the same transform JPEG uses for images) and merges the result into fewer tokens: that same second of motion takes 53. Training with it takes 5 times less compute, but inference writes the tokens one by one, about 750 ms a string, where flow-matching π0 takes under 100 ms.

*Flow matching: noise → 10 steps → 50 actions · action expert 300 million parameters · FAST: 700 → 53 tokens*

## 08 Slow and fast

A large model knows a lot but is slow; an arm needs speed. Several systems split the two. Gemini Robotics’ backbone runs in the cloud and answers in under 160 ms; a local decoder on the robot turns its output into strings of actions and covers the delay.

Figure’s Helix thinks 7 to 9 times a second with a 7-billion-parameter vision-language model, while an 80-million-parameter network puts out actions 200 times a second for 35 degrees of freedom. NVIDIA’s GR00T N1 has a slow part at 10 a second and a fast part at 120.

π0.5 writes its thinking as text: at each step it first names the next small task, such as “pick up the cutting board”, then gives a string of 50 actions. Gemini Robotics 1.5 also thinks before it acts, and shows its thinking. On the bench the display changes small tasks slowly, and the arm keeps moving.

*Helix: thinks 7⁠–⁠9 times a second, acts 200 · GR00T N1: 10 / 120 · Gemini Robotics backbone: \< 160 ms*

## 09 Where the data comes from

Robot data is taught by people, one demonstration at a time. RT-1’s roughly 130,000 demonstrations were recorded by operators with two VR controllers, on 13 robots, over 17 months. Gemini Robotics recorded for 12 months on a fleet of ALOHA 2 two-arm robots, thousands of hours. π0 used about 10,000 hours of its makers’ own data, from 7 robot configurations and 68 tasks.

One lab’s data is not enough, so labs pooled theirs. Open X-Embodiment (2023) gathered 60 datasets from 21 institutions: 22 kinds of robot, over a million trajectories, 527 skills. Every robot’s actions were written as the hand’s 7 quantities, but the coordinate frames were not aligned, so the same numbers can mean very different motions on different robots.

Other robots’ data helps. On skills that appear only in another robot’s data, RT-2-X, trained with the pooled data, succeeded 75.8 % of the time; RT-2 had managed 27.3 %, about 3 times less. OpenVLA used 970,000 of the demonstrations and trained for 14 days on 64 A100s.

*Open X-Embodiment: 22 kinds of robot · over a million trajectories · 60 datasets · 21 institutions*

## 10 Things it never saw

A cup it has never seen, a different tablecloth, another room: does the model still work? In RT-2’s tests, with unseen objects, backgrounds and rooms, RT-1 succeeded 32 % of the time on average and RT-2 62 %; on tasks they had seen, both did about the same, around 92 %.

π0.5 tidied kitchens and bedrooms in three real homes it never saw in training, 10 tries a task. Trained on data from about 100 different homes, it came close to a model trained in the very homes it was tested in.

These numbers don’t compare across papers. Each uses its own robots, its own tasks and its own scoring: some count only success or failure, some give credit for the steps done; often each task is tried only 5 to 20 times. The real world has no shared test site yet.

*RT-2, unseen scenes: 62 % (RT-1 32 %) · seen tasks: RT-1 92 %, RT-2 91⁠–⁠93 %*

## 11 What it gets wrong

RT-2’s paper lists what it does badly: grasping an object by a particular part (a handle, say), new motions not in its data (wiping with a towel), and dexterous, precise moves (folding a towel). OpenVLA typically succeeds less than 90 % of the time on the tasks tested. On the bench, this grasp closes on air.

Long tasks are harder. RT-1’s paper notes that success on long tasks falls exponentially with their length, so the success of every step matters. π0.5 sometimes gets distracted and opens and closes the same drawer again and again; when its own arm blocks the view, it can’t see the spill it should wipe.

Safety can’t be left to the model either. The Gemini Robotics report says such models must sit on top of safety-critical low-level controllers; its authors also released the ASIMOV datasets, which check whether a model knows things like “a soft toy must not be placed on a hot stove”. And data is costly: thousands of hours of people demonstrating.

*RT-2 does badly: handles, wiping, folding towels · OpenVLA: typically under 90 % · safety in the low-level controllers*

## 12 Taken apart

The bench comes apart in eight pieces: the arm, the gripper with its wrist camera, the depth camera on its mast, the controller (the model), the scales on its sloped face, the tape, the cup and cloth, and the riser plate under the arm.

The arm has five revolute joints: the base’s turn, the shoulder, the elbow, the wrist’s pitch and the wrist’s roll, with a parallel gripper at the end. The page turns each step’s hand position into joint angles with closed-form inverse kinematics.

Each scale has 256 bins, a short tick every 8 and a number every 64. Each row of the tape is one action’s 7 tokens.

*Arm · gripper · depth camera · controller · scales · tape · cup and cloth · riser plate*

## 13 Bench

On the bench you can change the instruction, how many actions each inference writes, and how long one inference takes. The arm plays actions at 50 a second; at each inference it stops and waits, then plays the whole string, then takes the next picture.

The readouts: the share of time the arm stands waiting, how many new images it looks at a second, and the longest it moves without a new look. With 1 action per inference the arm mostly waits; with 50 it hardly stops, but each look must last it a whole second.

The tokens on the riders and the tape are still written in RT-2’s format; the path is planned by the page.

*Instruction · actions per inference · one inference · readouts: time waiting, looks a second, longest without a look*

**Try this**: Raise the actions per inference from 1 to 50, then set one inference to 250 ms. How much less does the arm stand waiting, and how long does it go at most without a new look?

## What the model leaves out

There is no trained model on the bench. A simple planner makes the arm’s path, which is then written as tokens in RT-2’s format: each quantity cut into 256 equal bins over ranges the page chose (a real model takes them from its training data: RT-2 the bounds of each quantity, OpenVLA the 1st and 99th percentiles). The controller is a picture: RT-2’s large model actually ran in the cloud, OpenVLA and π0 on one GPU. The camera’s view is rendered from the camera’s position on the bench; the patch size follows no particular model. The arm is a generic five-axis arm, not any real robot; its joint angles come from closed-form inverse kinematics, gripper pointing down. The bench assumes the arm stops while the model thinks (synchronous execution); it does not show schemes that compute the next string while moving.

## Easy to get wrong

“50 Hz” is the beat of the actions, not of the model: π0 recomputes only every 0.5 seconds. RT-2’s 62 % against 32 % is the average over unseen scenes; on seen tasks it was no better. Success rates from different papers don’t compare: robots, tasks, scoring and the number of tries all differ. In Open X-Embodiment, RT-1-X’s “50 % higher” is a relative gain, and only on a few small datasets; on the large ones it did worse. These models learn to use the motions they have in new scenes; they don’t pick up new motions from nowhere. FAST makes training faster and inference slower. The companies’ “firsts” are their own claims.

## Common questions

### Why start a robot model from a vision-language model?

The web holds billions of images and captions; robot data comes nowhere near: even the largest robot datasets have 100,000 to a million demonstrations.

See: [03 Words in](https://world.lapla.si/en/vla/#words)

### How does RT-2 write robot actions as tokens?

RT-2 writes them as text. An action has 7 continuous quantities: how far the hand moves in x, y and z, how far it turns about three axes, and how wide the gripper opens.

See: [04 Actions as tokens](https://world.lapla.si/en/vla/#tokens)

### What does π0’s “50 Hz” mean?

π0 writes 50 actions at a time, 1 second at 50 a second. One pass takes 73 ms on an RTX 4090, while the arm plays only 3.65 actions.

See: [06 A chunk at a time](https://world.lapla.si/en/vla/#chunks)

### What do vision-language-action models still get wrong?

RT-2’s paper lists what it does badly: grasping an object by a particular part (a handle, say), new motions not in its data (wiping with a towel), and dexterous, precise moves (folding a towel).

See: [11 What it gets wrong](https://world.lapla.si/en/vla/#limits)

## Sources

1. [Brohan et al. (2022): RT-1: Robotics Transformer for Real-World Control at Scale](https://arxiv.org/abs/2212.06817)
2. [Brohan et al. (2023): RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://arxiv.org/abs/2307.15818)
3. [Google DeepMind (2023-07-28): RT-2: New model translates vision and language into action](https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/)
4. [Open X-Embodiment Collaboration (2023): Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://arxiv.org/abs/2310.08864)
5. [Kim et al. (2024): OpenVLA: An Open-Source Vision-Language-Action Model](https://arxiv.org/abs/2406.09246)
6. [Zhao et al. (2023): Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ALOHA, ACT)](https://arxiv.org/abs/2304.13705)
7. [Black et al. (2024): π0: A Vision-Language-Action Flow Model for General Robot Control](https://arxiv.org/abs/2410.24164)
8. [Pertsch et al. (2025): FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://arxiv.org/abs/2501.09747)
9. [Physical Intelligence (2025): π0.5: a Vision-Language-Action Model with Open-World Generalization](https://arxiv.org/abs/2504.16054)
10. [Gemini Robotics Team, Google DeepMind (2025): Gemini Robotics: Bringing AI into the Physical World](https://arxiv.org/abs/2503.20020)
11. [Google DeepMind (2025-09-25): Gemini Robotics 1.5 brings AI agents into the physical world](https://deepmind.google/blog/gemini-robotics-15-brings-ai-agents-into-the-physical-world/)
12. [NVIDIA (2025): GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://arxiv.org/abs/2503.14734)
13. [Figure AI (2025-02-20): Helix: A Vision-Language-Action Model for Generalist Humanoid Control](https://www.figure.ai/news/helix)

## Data

The arm, its tasks and the action tokens on the panel and the tape are the page’s own: a planned gripper path written in the format RT-2 uses (seven dimensions, 256 equal bins each), not the output of a trained model (src/topics/vla/science.js). The camera’s view is rendered from the camera on the mast; its grid of patches is a picture, not any model’s patch size. Material presets, parts and the softbox studio of the site’s model kit are ported from AirsupHQ/machine-skill (MIT, © 2026 Air Supply Technologies Inc.).

## Licence CC BY-NC 4.0

The text, pictures and models on this page are licensed CC BY-NC 4.0: free to use and adapt for teaching and other non-commercial purposes, with credit to world.lapla.si. Third-party material listed above keeps its own licence.

https://creativecommons.org/licenses/by-nc/4.0/deed.en

## Cite this plate

Laplace’s Mini World, Plate F44, “Vision-Language-Action Models”, https://world.lapla.si/en/vla/, CC BY-NC 4.0, updated 2026-10-24.