I spend most of my days working with LLMs and building backend systems. So when I first heard about Vision-Language-Action (VLA) models like OpenVLA, π₀, and Octo, my brain immediately slotted them into a familiar box: “multimodal LLM, different output head, done.”

Turns out that’s roughly as accurate as saying a Formula 1 car is just a sedan with a bigger engine.

The thing that broke my mental model was latency. When you’re streaming tokens from a chatbot, an extra 200 ms is barely noticeable. Maybe the cursor blinks a beat longer. When you’re controlling a robot arm that needs to react at 30 Hz, 200 ms means it’s already overshot, crashed into the table, or yeeted whatever it was holding across the room. You can’t retry a physical collision. There’s no Ctrl+Z in meatspace.

I spent a few weeks going deep on the papers, and I want to share what clicked for me: how we got here, how these systems actually work, and why turning language into physical torque is a fundamentally harder problem than I’d appreciated.

How Robotics Got Here

Before diving into VLAs, it’s worth understanding the path that led to them. Each era solved something and left a gap the next one tried to fill.

flowchart TD A["Classical Control & Motion Planning\n(1980s-2010s)\nInverse Kinematics, PRM, RRT"] --> B["Deep Reinforcement Learning\n(2015-2020)\nEnd-to-End, Sim-to-Real, Narrow Tasks"] B --> C["Vision-Language Models\n(2021-2023)\nSemantic Scene Understanding"] C --> D["Vision-Language-Action Models\n(2023-Present)\nToken-to-Torque Policies"]

The Classical Era

For decades, robotics was all hand-engineered pipelines: perception module talks to state estimator, state estimator feeds a path planner (RRT, PRM), planner hands off to a PID controller. Each piece did its job, and the whole thing worked. Right up until something changed. Move a mug two centimetres to the left, change the lighting, introduce an object the system hadn’t been told about, and the whole pipeline would faceplant. It was like a house of cards built on the assumption that the world stays perfectly still.

Deep RL and Imitation Learning

Around 2015, things like QT-Opt and DeepMimic showed that you could skip the handcrafted pipeline entirely and just learn from pixels to motor torque, end-to-end. First time I read those papers, it felt like sorcery.

But it didn’t generalise. At all. Train a policy to pick up a red apple and it’d stare blankly at a green one. Every new object, every new environment, meant thousands more hours of training. And getting anything learned in simulation to work on real hardware required absurd amounts of domain randomisation. The gap between sim and real was brutal.

The VLA Shift

VLAs come at the problem differently. Instead of learning everything from scratch, they piggyback on the massive multimodal pre-training that models like PaLM-E and LLaVA have already done. The foundation model already understands spatial relationships, knows what a “mug” is, can parse instructions. You fine-tune that on real robot trajectory data (datasets like Open X-Embodiment) and suddenly you’ve got something resembling generalised physical intuition. It’s not perfect, but the leap in flexibility is hard to overstate.

The Part That Blew My Mind: Fast vs. Slow Thinking

This is the section that really rewired my understanding.

I’m a big fan of Kahneman’s Thinking, Fast and Slow. The core idea: humans have two cognitive systems. System 1 is fast and instinctive. You duck when someone throws a ball at your face. System 2 is slow and deliberate. You sit and calculate 17 × 24.

When I first thought about VLAs, my approach was pure System 2: tokenise joint angles, pack them into the LLM context with camera frames, run autoregressive inference. What could go wrong?

Everything, it turns out. If your whole pipeline runs through a 7B model doing full attention at every step, you’re stuck at maybe 2 Hz. The robot moves like it’s wading through treacle. It can’t react to anything unexpected because by the time it’s “thought” about a disturbance, the moment is long gone.

The clever trick in modern VLAs is that they split things exactly like Kahneman would:

graph TB subgraph System2 ["System 2: The Slow Planner (~1-2 Hz)"] A["Camera Feed"] --> B["VLM Backbone\n2B-7B Params"] C["Instruction: 'Pass the mug'"] --> B B --> D["Goal Latent Embeddings"] end subgraph System1 ["System 1: The Fast Policy (~30-50 Hz)"] D --> E["Action Expert Head\n30M-300M Params"] F["Proprioception / Telemetry"] --> E E --> G["Action Trajectory Chunks"] end G --> H["Robot Actuators"]

The slow planner (System 2) is a big VLM, 2B to 7B parameters, running in a background thread at 1-2 Hz. It’s doing the heavy lifting: understanding the scene, parsing the instruction, figuring out what to do. “The red mug is behind the plate, so I should reach around.”

The fast policy (System 1) is a lightweight action head, 30M to 300M parameters, screaming along at 30-50 Hz. It takes the goal embedding from the planner, combines it with real-time sensor data from the robot’s joints, and outputs continuous motor commands. This is the reflex layer. It doesn’t need to understand the scene; it just needs to smoothly execute the plan while reacting to micro-perturbations.

This architecture is the single biggest “aha” for me. You don’t want your biggest, smartest model controlling every motor twitch. You want it thinking about what to do, and a smaller specialist worrying about how to move.

Papers I’d Actually Recommend Reading

There are a ton of papers in this space, but these are the ones that built my understanding:

RT-1 & RT-2 (Google DeepMind, 2022-2023): RT-1 proved transformers could do real-world robot control at scale. RT-2 was the breakthrough: take a pre-trained VLM, fine-tune it on robot trajectories, and watch it follow instructions like “pick up the extinct animal” by correctly grabbing the toy dinosaur. That was the moment this stopped feeling like a research curiosity to me.

ACT (Zhao et al., Stanford, 2023): Introduced action chunking and temporal ensembling. Sounds dry, but it’s arguably the most important practical contribution. Predicting one action at a time causes compounding drift. Predicting a chunk of future actions and blending overlapping predictions is what makes these policies actually usable.

Octo & OpenVLA (2024): Octo is a diffusion-based policy trained on 800k+ trajectories across 25 robot types. OpenVLA slaps a 7B Llama 2 backbone on the problem and runs on consumer GPUs with 4-bit quantisation. These are the models that made me feel like I could actually try this stuff without a robotics lab budget.

π₀ (Physical Intelligence, late 2024): This one is architecturally the most exciting. It ditches discrete action tokens entirely in favour of flow matching, learning continuous vector fields via ODEs. The result is a system that can do dexterous, multi-arm manipulation with fast tactile feedback. Reading this paper felt like seeing the future.

Tokens vs. Noise vs. Flows: The Action Output Problem

This bugged me for a while. LLMs output discrete tokens. Robot joints need continuous signals: angles, velocities, torques. How do you get from one to the other?

There are three schools of thought right now:

Approach The Idea Trade-off
Discretised Binning Chop continuous joint angles into 256 bins per axis, treat them as vocabulary tokens. (RT-1, RT-2, OpenVLA) Dead simple to train. But quantisation error kills fine motor precision, and sequence length explodes with more joints.
Diffusion Heads Start with noise, iteratively denoise conditioned on visual latents until you get a trajectory. (Diffusion Policy, Octo, ACT) Beautifully handles multimodal actions (there’s often more than one good path). But the denoising steps add latency.
Flow Matching Learn a continuous vector field that maps noise to trajectories in one smooth pass using ODEs. (π₀, SmolVLA) Fewer steps than diffusion, continuous outputs. But you need to be careful with the vector field design.

No silver bullet. Each buys you something and costs you something else.

Why Action Chunking Matters More Than I Expected

I initially assumed these models worked step-by-step: see state, predict next action, execute, repeat.

That’s a trap. Even tiny errors compound. A 0.5° mistake on step 1 becomes a 15° drift by step 30, and now your robot is reaching for thin air.

The fix is action chunking: instead of predicting one step, the model predicts a whole trajectory, 30 to 50 future actions, in a single forward pass. As the robot executes, overlapping prediction windows get blended together with exponential decay weighting (temporal ensembling).

The blending smooths out jitter and prevents the robot from vibrating itself to death as fresh predictions roll in. It’s one of those ideas that seems obvious in hindsight, but it took a solid paper (ACT) to nail down the details.

Putting It Together in Code

To make the dual-system architecture concrete in my head, I sketched out a toy PyTorch model. It’s simplified to the point of being wrong in many ways, but it captures the core split:

import torch
import torch.nn as nn

class DecoupledVLA(nn.Module):
    def __init__(self, slow_planner_vlm, fast_action_expert):
        super().__init__()
        # System 2: heavy VLM, runs async at ~1-2 Hz
        self.planner = slow_planner_vlm
        # System 1: lightweight policy, runs real-time at ~30-50 Hz
        self.action_expert = fast_action_expert

    def get_goal_representation(self, image: torch.Tensor, prompt: str) -> torch.Tensor:
        """Background thread. Slow. Understands the world."""
        with torch.no_grad():
            goal_latent = self.planner.extract_features(image, prompt)
        return goal_latent

    def step_actuation(self, proprio_state: torch.Tensor, goal_latent: torch.Tensor) -> torch.Tensor:
        """Main control loop. Fast. Moves the joints."""
        action_chunk = self.action_expert(proprio_state, goal_latent)
        return action_chunk  # [Batch, Horizon, DoF]

The planner runs in a background thread, updating the goal every second or so. The action_expert runs on every control tick, consuming the latest goal embedding plus fresh proprioceptive data. Two loops, two speeds, one robot.

What I’m Still Thinking About

I’ve got more questions than answers at this point, but two keep nagging at me:

Will universal policies ever just work? Cross-embodiment models trained on dozens of robot types are impressive, but every real motor has its own backlash, friction, and sensor noise. I’m not yet convinced you can ship a single checkpoint that works out-of-the-box on arbitrary hardware without at least some calibration fine-tuning.

How fast can the reflex layer bail out? If someone bumps into the robot mid-trajectory, System 1 needs to interrupt immediately. But the current action chunk was planned assuming a clear path. How quickly can the fast policy override its own predictions? And what happens to System 2’s world model when the scene changes out from under it?

I’m planning to dig deeper into both of these, probably starting with some hands-on experiments once I get access to a decent sim environment. If you’re working in this space, I’d genuinely love to hear what you’re seeing. Hit me up on any of my socials.