The question I get asked most when I explain what a robot foundation model actually does is some version of: "wait, so it's a chatbot that also moves the arm?" That's closer to correct than people expect, and it's the fastest way into understanding a vision-language-action (VLA) model — a model that takes a camera image and a natural-language instruction as input and produces a robot action as output, built by taking a model that already understands images and language and teaching it to also speak "motor command." I covered the landscape framing in the Physical AI overview; this piece is the mechanism — how a VLA model is actually built, trained, and run, and where it still falls over.
What is the core idea behind a VLA model?
A VLA model takes a pretrained vision-language model (VLM) — something already trained to look at an image and answer a question about it, or follow a language instruction, at internet scale — and adds an action head: a small additional component that turns the model's internal representation into a robot action instead of (or in addition to) a text response. The VLM backbone contributes something a robotics-only model would take enormously more real data to learn on its own: what objects look like, what language means, how the two relate. The action head is what actually has to be learned from robot-specific data, because "what a mug looks like" and "how to close your gripper around this specific mug without crushing it or dropping it" are very different problems, and only the second one requires physical demonstration data.
This is the single idea to hold onto: a VLA model isn't learning robotics and vision and language all from scratch off the same small pool of demonstrations. It's learning robotics on top of vision and language it already has, which is why VLA models need far less task-specific data than training a policy from a blank network would.
graph TD
IMG["Camera frame"] --> VLM["Pretrained vision-language backbone
(internet-scale prior)"]
LANG["Language instruction
e.g. 'pick up the red cup'"] --> VLM
VLM --> HEAD["Action head
(discrete tokens OR flow/diffusion)"]
HEAD --> ACT["Robot action
joint angles / end-effector pose"]
ACT -.->|"executed, new frame observed"| IMG
The shared shape of every VLA model. The backbone contributes visual and semantic understanding pretrained on internet-scale data; the action head is the part specialized on robot demonstration data. The two design choices that differentiate real systems are (1) what data trained the backbone and (2) how the action head turns its output into an actual motor command — which is the split covered below.
How does a model output a continuous motor command from a token-based architecture?
This is the part that isn't obvious, and there are two genuinely different answers in production today.
Discrete action tokens: treat motor commands like language
Google DeepMind's RT-2 took the more conceptually direct route: discretize the continuous action space (each dimension of the robot's movement — x, y, z, rotation, gripper open/close) into a fixed number of bins, and represent each bin as a token in the model's existing vocabulary. The model then predicts actions the exact same way an LLM predicts the next word — autoregressively, one token at a time, reusing the entire existing transformer decoding machinery unmodified. This is the elegant part of the design: you don't need a new architecture, a new loss function, or a new training pipeline. You just add "action tokens" to the vocabulary and keep doing next-token prediction.
OpenVLA, the open-source 7B-parameter model out of Stanford and collaborators, follows the same discrete-token pattern — it fuses visual features from DINOv2 and CLIP with a Llama-2 language backbone, and outputs the same kind of discretized action tokens, trained on the Open X-Embodiment dataset (roughly a million robot manipulation episodes contributed by more than twenty research labs, spanning around twenty different robot embodiments). Open X-Embodiment matters as much as any single model architecture — it's the first dataset that gave the field something resembling a shared, multi-robot training corpus, the closest robotics has to its own "internet-scale" text corpus, even though it's still many orders of magnitude smaller.
Flow matching / diffusion: predict a continuous trajectory directly
Physical Intelligence's π0 (pi-zero) model takes the other route: instead of discretizing the action space and predicting it token by token, it uses flow matching — a technique from the same family as diffusion models — to directly generate a continuous sequence of future actions in one denoising process, conditioned on the vision-language backbone's understanding of the scene and instruction. The output isn't a string of discrete tokens decoded one at a time; it's a smooth trajectory produced in a small, fixed number of refinement steps, which is dramatically faster at inference and produces motion that's naturally smoother because it was never quantized into bins in the first place.
| Discrete tokens (RT-2, OpenVLA) | Flow matching / diffusion (π0) | |
|---|---|---|
| Action representation | Quantized bins, one token per dimension per timestep | Continuous vector, generated directly |
| Decoding | Autoregressive, one token at a time | Iterative refinement, fixed small step count |
| Infrastructure reuse | Nearly free — same decoder as an LLM | Requires a diffusion/flow training recipe |
| Inference speed | Slower — token-by-token has a latency floor | Faster — few refinement steps, higher control frequency |
| Motion quality | Can be jerky at bin boundaries | Naturally smooth, continuous |
| Best fit | Simpler tasks, slower-reflex settings, easiest to stand up | Dexterous, high-frequency, real-time manipulation |
Physical Intelligence also released π0-FAST, an autoregressive variant using a different action tokenizer, which is itself a tell about how unsettled this trade-off still is — even the same lab shipped both an autoregressive and a flow-matching variant rather than declaring one approach dead. Reported figures put π0's action generation running at roughly 50Hz, which is the kind of control frequency that matters for tasks with fast, continuous feedback loops (folding fabric, inserting a plug) — a token-by-token autoregressive decoder run at that frequency starts to hit a real latency wall.
Why does action chunking matter?
Action chunking means predicting a short sequence of several future timesteps' worth of actions in a single forward pass, instead of predicting one action, executing it, observing the result, and predicting the next action one step at a time. It sounds like a minor implementation detail and it is actually load-bearing for two reasons. First, latency: if your model has to run a full forward pass before every single motor command, your control frequency is capped by inference time — chunking amortizes that cost across several timesteps of action at once. Second, and less obvious: chunking measurably improves smoothness and stability, because predicting one step at a time lets small per-step errors compound and lets the policy drift into stalling or jittering behavior near difficult transitions (contact, occlusion) — predicting a short coherent sequence gives the model a chance to commit to a trajectory rather than dithering step by step. Most modern VLA systems, whether autoregressive or flow-based, use some form of chunking; the flow-matching approach used by π0 is a particularly natural fit for it since it's already generating an extended trajectory in one pass.
How are these models actually trained?
The training recipe has two stages that mirror the architecture split described above. First, the vision-language backbone is pretrained (or reused pretrained) on internet-scale image-text and video data — this is where the model learns what objects, scenes, and instructions mean, entirely without any robot involved. Second, the whole model (or just the action head, depending on the approach) is fine-tuned on robot demonstration data — large offline datasets of teleoperated trajectories, of which Open X-Embodiment is the most-cited multi-institution example, alongside proprietary datasets collected by labs like Physical Intelligence (reportedly on the order of ten thousand-plus hours across multiple robot platforms and dozens of distinct tasks). The demonstrations are typically collected via teleoperation — a human operator directly driving the robot's end effector through a task while every observation and action gets logged — because it's currently the most reliable way to get correctly-labeled (observation, action) pairs at any real volume, expensive as it still is per hour compared to a scraped web page.
# conceptual shape of one VLA training example
# (real datasets like Open X-Embodiment package this per-episode, per-timestep)
example = {
"image": camera_frame, # RGB observation at time t
"instruction": "pick up the red cup", # natural language task
"action": {
"arm_delta": [0.02, -0.01, 0.05], # continuous end-effector delta
"gripper": 0.0, # 0 = open, 1 = closed
},
"robot_embodiment": "franka_panda", # which robot collected this
}
# discrete-token models quantize `action` into vocabulary tokens;
# flow-matching models train the action head to denoise it directlyBe honest about what VLA models are still bad at. Three limitations show up in nearly every serious deployment. First, they're still data-hungry relative to how little robot demonstration data exists in the world — the internet-scale prior helps enormously with perception and language grounding, but the action policy itself is still bottlenecked by expensive teleoperation data. Second, generalization across robot embodiments and genuinely novel objects is imperfect — a model trained mostly on one arm and gripper configuration degrades on a different robot's kinematics, and "never seen this object" performance is real but inconsistent, not the seamless transfer the demo reels imply. Third, inference latency is a hard engineering constraint, not a footnote — a model that takes 200ms per forward pass is unusable for a task that needs a reflexive correction in 20ms, and that's exactly why the discrete-token-vs-flow-matching choice above isn't academic; it's the decision that determines whether your robot can catch a falling object or only pick up ones that already stopped moving.
How do you choose between the two approaches in practice?
If you're standing up a research or pilot system and want to reuse existing LLM serving infrastructure with minimal new plumbing, a discrete-token model like OpenVLA is the lower-friction starting point — you already know how to serve, batch, and fine-tune an autoregressive transformer. If your task genuinely needs high-frequency, dexterous, continuous control — anything involving fabric, fine manipulation, or fast reactive corrections — the inference-speed and smoothness advantages of a flow-matching approach like π0 are hard to argue with, and the training-recipe cost of adopting diffusion-style training is a one-time tax rather than a per-inference one. In both cases, the actual bottleneck you'll hit first in production isn't the architecture — it's the volume and diversity of real demonstration data you can afford to collect for your specific robot and task.
What to carry away
A vision-language-action model works by attaching an action-producing head to a vision-language backbone already pretrained at internet scale, so the model only has to learn the robotics-specific part — mapping perception and instruction to motor commands — from the comparatively tiny pool of real demonstration data. RT-2 and OpenVLA represent that action head as discrete tokens decoded autoregressively, reusing LLM infrastructure at the cost of inference speed; Physical Intelligence's π0 represents it as a continuous flow-matching output, trading a new training recipe for faster, smoother control. Action chunking — predicting several timesteps at once — is what makes either approach fast and stable enough to run on real hardware. None of this closes the gap entirely: these models are still data-hungry, generalize imperfectly across robot bodies and novel objects, and carry inference-latency constraints that are a real engineering wall, not a rounding error. The pipeline that feeds these models — and why simulation had to become the answer to the data-hunger problem — is the subject of the next piece in this set, on sim-to-real and digital twins.