Words In, Joints Out

Vision-language-action models, from RT-2 to π0.7 — the one shape nearly all of them share, which ones you can download and under what licence, what they ask of you before they will move your robot, and what could be checked from a machine with no GPU

Author

Behzad Samadi

Published

September 13, 2026

Say “put the mug on the shelf” to a robot arm that has never seen your kitchen, and watch it do it. Three years ago that was a Google demo; today it is a category with a name — vision-language-action — a dozen serious entrants, and a $100 arm you can train one on. The previous article mapped NVIDIA’s stack and found GR00T sitting at the top of it as “the robot brain.” This one is about that layer across the whole field: what a VLA is, why they all look alike inside, which ones are actually open, and what each wants from you.

It was written the same way as the last one. Everything that could be read at the source was — the projects’ repositories, their licence files, the package index — and one thing was run: the framework most of the open models ship in, installed here and made to state each model’s shape from its own code. Nothing was downloaded from the model hub, because from this machine it could not be; the article says which numbers are the projects’ claims and which are this machine’s.

NoteTL;DR
  1. One shape. A pretrained vision-language model reads the camera and the sentence; an action head turns its hidden state into a chunk of 16–50 future joint targets, executed before the model is asked again. RT-2 wrote actions as text tokens; almost everything since 2024 uses a separate flow-matching or diffusion head instead.
  2. “Open” is three questions. Is the code open, are the weights open, and do the weights exist? π0.5 and SmolVLA are Apache-2.0 end to end; OpenVLA’s weights carry Llama’s licence; GR00T’s carry NVIDIA’s; π0.7, Helix and Gemini Robotics 1.5 have no public weights at all.
  3. LeRobot is the through-line. Nine of the eleven VLAs named here ship as policies in Hugging Face’s LeRobot (0.6.1, 3 August 2026, Python 3.12 only) — the same training command, the same dataset format, the same SO-101 arm.
  4. The price of entry is a robot, fifty demonstrations and an afternoon of GPU. Fine-tuning SmolVLA is “~4 hrs on a single A100”; the small VLAs want 10–16 GB of VRAM, the large ones 24–40. On a CPU, LeRobot’s own guide says: “Don’t train.”
  5. The frontier is closed and the fast end is open. Physical Intelligence, Figure and Google DeepMind keep their newest models; the open field has answered with smaller, faster ones — 450M (SmolVLA), 0.77B (EVO-1), 0.9B (X-VLA) — and with reasoning traces you can draw before the arm moves (MolmoAct2).

The shape they all share

Strip the names away and a VLA is two parts and a habit.

The backbone is a vision-language model somebody else trained: PaliGemma for π0, Llama-2 with DINOv2 and SigLIP for OpenVLA, SmolVLM2 for SmolVLA, Cosmos-Reason2 for GR00T, Qwen2.5-VL for EO-1 and WALL-OSS, InternVL3 for EVO-1, Florence-2 for X-VLA. It brings the part robotics could never afford to learn from robot data alone — what a mug is, what “the shelf” means, what “gently” implies. The robot-specific training teaches it to look at a gripper camera instead of a photo.

The action head turns the backbone’s hidden state into numbers a motor controller can use. RT-2, in 2023, did this the blunt way: each of seven end-effector degrees of freedom into 256 bins, written as integers, so an action was a string like “1 128 91 241 5 101 127” and the language model simply spelled it out. OpenVLA kept the trick. Everything since π0 has mostly moved to a second, smaller network — a “flow-matching” or “diffusion” expert — that reads the backbone’s key-value cache and denoises a whole block of continuous actions at once.

The habit is the chunk. No VLA runs at control rate. It predicts a chunk of future actions and lets the controller play them out while it thinks about the next one. Read from the installed LeRobot 0.6.1 on this machine, the defaults are: 50 steps for SmolVLA, π0 and π0.5 and EVO-1; 40 for GR00T N1.7; 32 for X-VLA and WALL-OSS; 30 for MolmoAct2; 16 for EO-1. At a 30 Hz controller, a chunk of 50 is 1.7 seconds of motion decided in one forward pass — which is why “asynchronous inference” and “real-time chunking” are the words on every 2025 release: start computing the next chunk before the current one runs out, and splice.

Figure’s Helix makes the same split with two networks instead of a chunk: a ~7B vision-language “System 2” at 7–9 Hz and an 80M “System 1” visuomotor policy at 200 Hz. Different mechanism, same admission — the model that understands the sentence is too slow to close the loop, so something faster does.

The lineage

  • 2023 — RT-2 and Open X-Embodiment (Google DeepMind). RT-2 coined the term and showed that a 55B vision-language model fine-tuned to emit action tokens generalised in ways a robot-only policy did not. Open X-Embodiment pooled “1M+ real robot trajectories spanning 22 robot embodiments” from 21 institutions and trained RT-X on it: the first evidence that other robots’ data helps yours. Both closed as models; the dataset is open.
  • 2024 — OpenVLA (Stanford, Berkeley and others) and π0 (Physical Intelligence). OpenVLA, 7B, trained on 970K Open X-Embodiment trajectories, beat RT-2-X “by 16.5% absolute success rate across 29 evaluated tasks” with an eighth of the parameters, and was downloadable. π0 introduced the flow-matching action expert on a PaliGemma backbone, trained on data from seven robots, and set the shape everyone now uses.
  • 2025 — the crowd. Physical Intelligence open-sourced π0 and π0.5; Hugging Face released SmolVLA at 450M and the SO-101 arm at $100; NVIDIA shipped GR00T N1; Figure showed Helix; Google DeepMind put Gemini Robotics On-Device in the hands of trusted testers and, in September, Gemini Robotics 1.5; Ai2’s MolmoAct made the model draw its plan; X-VLA, EO-1, EVO-1 and WALL-OSS arrived from universities and start-ups, most of them straight into LeRobot.
  • 2026, so far. Helix-02 took over the whole humanoid body in January; GR00T N1.7 and π0.7 in April; MolmoAct2 in May; Gemini Robotics 2, On-Device 2 and ER 2 on 30 July; LeRobot 0.6 in July. Three of the eleven models below changed in the last twelve weeks.

The open ones

“Open” needs three answers, and the table gives all three.

Model Who, when Backbone → head Size Code Weights
OpenVLA Stanford et al., June 2024 Llama-2 7B + DINOv2/SigLIP → action tokens 7B MIT Llama Community License
π0 / π0-FAST / π0.5 Physical Intelligence, 2024–25 PaliGemma (Gemma 2B) → Gemma 300M flow expert ~3.3B Apache-2.0 Apache-2.0 (openpi)
SmolVLA Hugging Face, June 2025 SmolVLM2-500M (16 layers) → ~100M flow expert 450M Apache-2.0 Apache-2.0
GR00T N1.7 NVIDIA, April 2026 Cosmos-Reason2-2B → flow-matching DiT 3B Apache-2.0 NVIDIA Open Model License
X-VLA Tsinghua-led, Oct 2025 (ICLR 2026) Florence-2 family → flow matching, soft prompts 0.9B Apache-2.0 Apache-2.0
EO-1 2025 Qwen2.5-VL-3B → flow matching ~3B open open
EVO-1 MINT-SJTU, 2025 InternVL3-1B → flow matching 0.77B open open
MolmoAct2 Ai2, May 2026 Molmo2-ER → flow expert, with a visual trace — Apache-2.0 Apache-2.0
WALL-OSS X Square Robot, Sept 2025 Qwen2.5-VL MoE → flow or FAST tokens — Apache-2.0 Apache-2.0

OpenVLA is the reference point papers still compare against, and the one whose licence footnote matters: MIT code, but the weights inherit Llama-2’s community licence. It is also the heaviest to run — bf16 with Flash Attention, “approximately 27GB” of GPU memory even for LoRA fine-tuning — and slow, one action token at a time, until the 2025 OFT recipe (parallel decoding, chunking, a continuous head) made it 26× faster and took its LIBERO score from 76.5% to 97.1%. Read it as history that still works.

π0 and π0.5 are the models the shape is named after. Physical Intelligence’s openpi repository ships pi0_base, pi0_fast_base and pi05_base, “pre-trained on 10k+ hours of robot data,” with fine-tunes for DROID, ALOHA and LIBERO, all Apache-2.0; inference wants “> 8 GB”, LoRA fine-tuning “> 22.5 GB”, full fine-tuning “> 70 GB”. Nothing after π0.5 is in it — see below.

SmolVLA is the one to try first. SmolVLM2-500M with only its first sixteen layers kept, a flow-matching expert three-quarters of the VLM’s width, 450M parameters in all, pretrained not on a lab’s fleet but on 481 community datasets — about 23,000 episodes, mostly from SO-100 arms in people’s homes — and it lifted real SO-100 success from 51.7% to 78.3% over training from scratch. Fine-tuning “for 20k steps will roughly take ~4 hrs on a single A100 GPU.” It is the only model in the table whose whole pipeline — arm, data, pretraining, fine-tuning — is meant for a person, not a company.

GR00T N1.7 is NVIDIA’s, covered in the previous article: 3B, a Cosmos-Reason2-2B backbone, 16 GB of VRAM to run and 40 to fine-tune, and the one licence in the table that is NVIDIA’s own rather than Apache’s.

The rest of the LeRobot shelf is where the field’s variety lives. X-VLA treats each robot as a task, encoded in 32 learnable “soft-prompt” embeddings, so one 0.9B model adapts to a new arm by tuning “only 1% of parameters (9M).” EO-1 and WALL-OSS put a 3B Qwen2.5-VL behind a flow head, WALL-OSS with a mixture of experts and a choice of continuous or FAST-token output. EVO-1 packs an InternVL3-1B backbone and a flow head into 0.77B, small enough to fine-tune on a modest GPU. MolmoAct2 is the interesting one: an “action reasoning model” that first emits depth tokens and a visual trace — integer image coordinates you can draw on the camera frame — and only then the actions, so you can see the plan before the arm commits to it; weights, data and code Apache-2.0, deployable out of the box on the SO-100.

The closed ones

The frontier is not on the shelf, and it is worth knowing what is not there.

Physical Intelligence stopped publishing weights after π0.5. π*0.6 (late 2025) used reinforcement learning to run an espresso machine through 13-hour shifts; π0.7 (16 April 2026) is “a steerable model with emergent capabilities,” prompted with language, metadata and visual sub-goals from a small world model, and its claim is compositional generalisation — an air fryer operated after “two fragmentary references during training.” Not open.

Figure’s Helix (February 2025) was the first VLA to drive a whole humanoid upper body — 35 degrees of freedom, wrists and fingers included — at 200 Hz from onboard GPUs, trained on “approximately 500 hours” of demonstrations. Helix-02 (January 2026) added a “System 0” locomotion network trained on a thousand hours of human motion in place of “more than 109,000 lines of hand-engineered C++,” and by May Figure was showing teams of robots on eight-hour shifts. Closed.

Google DeepMind’s Gemini Robotics is three products under one name. Gemini Robotics 1.5 (September 2025) is the VLA, the one that “thinks before taking action”; Gemini Robotics-ER is the embodied-reasoning model that plans, calls tools and orchestrates, and is the one you can actually use — ER 1.5 in the Gemini API since September 2025, ER 2 since 30 July 2026, when Gemini Robotics 2 extended the VLA to whole-body humanoid control and On-Device 2 promised adaptation to a new robot “with a few hours of training.” The VLAs go to early-access partners; the reasoning model goes to everyone. That split — reasoning open through an API, actions closed — may be the shape the closed half of the field settles into.

LeRobot, the through-line

Hugging Face’s LeRobot is not a model. It is the reason the open models are comparable at all: one lerobot-train command, one dataset format, one hardware guide, and a policy folder that on 13 September 2026 holds twenty entries — act, diffusion, eo1, evo1, fastwam, gaussian_actor, groot, lingbot_va, molmoact2, multi_task_dit, pi0, pi05, pi0_fast, rtc, smolvla, tdmpc, vla_jepa, vqbet, wall_x, xvla — behaviour cloning, VLAs, world models and reward models side by side. Apache 2.0. Version 0.6.1 shipped on 3 August 2026; 0.5.0, in March, raised the floor to Python 3.12.

Its dataset format is what “your data” means. A LeRobotDataset is Parquet for “low-dimensional, high-frequency signals (states, actions, timestamps)”, MP4 for “camera frames concatenated and encoded” per camera, and JSON for the schema and normalisation statistics; each frame carries observation.state, action, observation.images.<camera>, a timestamp and the task string. Record it by teleoperating the arm — a second arm, a gamepad, a phone — and you have what every model in the table trains on.

Its hardware guide is the only cross-model table anyone publishes. Peak VRAM at batch size 8: light behaviour cloning (ACT, VQ-BeT) 2–6 GB; diffusion 8–14 GB; “Small VLA” — SmolVLA — 10–16 GB, “RTX 4080+, L4, A10G”; “Large VLA” — π0, π0.5, X-VLA, WALL-OSS — 24–40 GB, “A100 40GB+”; “Multimodal” — GR00T, EO-1 — the same. Five epochs on 45k frames: ACT on an RTX 4090 in 30–60 minutes; SmolVLA on an L4 in 3–6 hours; π0 on an A100 in 4–8. ACT on an Apple-silicon Mac, 6–14 hours. And the line for a machine like the one this was written on: “Don’t train. Use Colab or rent a GPU.”

What ran here, with no GPU and no hub

The machine is the same GPU-less four-core container as last time; this time the model hub was blocked as well, so the experiment was: how much of a VLA can be verified from the code that runs it?

pip install "lerobot[smolvla]" on Python 3.11 took 4 min 13 s, wrote 8.3 GB, and installed 0.4.4 — the February release — because everything newer is >= 3.12 and pip silently takes the newest version the interpreter is allowed. The same trap as Isaac Sim, one article ago. On Python 3.12 with uv: 1 min 5 s, 4.9 GB, 0.6.1.

Then a short script imported the package (10 s) and instantiated every VLA’s configuration with its defaults. That is where the chunk sizes above come from, and the backbone names — HuggingFaceTB/SmolVLM2-500M-Video-Instruct with num_vlm_layers = 16; gemma_2b and gemma_300m for the π family; nvidia/GR00T-N1.7-3B; OpenGVLab/InternVL3-1B-hf; x-square-robot/wall-oss-flow. Ten of eleven configurations exist without a network. The eleventh, EO-1, calls the Hub in its constructor to fetch Qwen2.5-VL’s configuration and cannot be built offline — a small fact about where these models’ definitions actually live.

What did not run: any model. No weights, no forward pass, no timing. Every success rate and every hour-on-an-A100 in this article is the project’s number, not this machine’s; the notes say so line by line.

What a VLA wants from you

In the order you will meet them.

  • A robot with a camera, and a way to drive it by hand. The SO-101 — the 3D-printed six-axis arm Hugging Face announced in April 2025 “starting at $100” in parts, $300–500 assembled or built — is what SmolVLA was pretrained on and what MolmoAct2 deploys to “out of the box.” A second arm as a leader, or a gamepad, records the demonstrations.
  • Demonstrations, in the dozens. Gemini Robotics On-Device fine-tunes “with just 50–100 demos”; SmolVLA’s recipe is 20k steps over your dataset; the community models it was pretrained on average about fifty episodes each. Record the task the way you want it done, and record it varied.
  • A GPU for an afternoon, not a data centre. 10–16 GB for the small VLAs, 24–40 for the large; four hours of A100 for SmolVLA; Colab or a rented L4 if the machine on your desk is a Mac. Inference on the robot is cheaper than training — GR00T runs on a Jetson, Helix on two embedded GPUs — but nothing in this article runs on a CPU at control rate.
  • A sentence. The task string is a first-class input, stored per frame in the dataset. Write the instruction you will actually say.
  • The version check. python --version before pip install lerobot: 3.12 or newer, or you are six months behind and will not be told.

What to do this week

  • Read one policy configuration before reading one paper: lerobot/policies/smolvla/configuration_smolvla.py. The whole shape — backbone, layers kept, expert width, chunk — is thirty lines.
  • Install LeRobot on Python 3.12 and run python -c "import lerobot; print(lerobot.__version__)". If it says 0.4.x, your interpreter is too old.
  • Pick your first model by your GPU, not by its score. Under 16 GB: SmolVLA or EVO-1. Over 24 GB: π0.5. On a Jetson: GR00T. No GPU: read, and rent one for the afternoon you fine-tune.
  • Record ten demonstrations of one task on whatever you have — a simulated arm counts — in LeRobot’s format, and look at the Parquet. The dataset is the part of this you will spend your time on.
  • Keep the closed models in view, not in plan. ER 2 is in the Gemini API today; Gemini Robotics 2, π0.7 and Helix-02 are not coming to your arm. Design for the open shelf.

Sources

Research notes — the pages read, the PyPI records, the installed configurations, and the pages that could not be fetched — are in the accompanying folder.