Running a VLA in real time: the inference latency problem

A humanoid reaching for a falling cup does not get to think for a fifth of a second. The cup is moving, the contact is coming, and the hand has to arrive before the physics finishes without it. A robot's controller lives on a clock: it has to emit a fresh command every few tens of milliseconds, forever, or the motion tears. This constraint almost never shows up in a demo reel, and it is the one that decides whether a policy ships.

Here is the tension. The strongest manipulation policies today are large vision-language-action models: transformers with billions of parameters that read camera frames and a language instruction and return motor commands. A single forward pass through one of them takes real, measurable time, often on the order of 100 milliseconds on deployment hardware. The world, meanwhile, keeps changing at tens of hertz. Run the model naively, once per control step, and the robot stutters: it waits, acts on stale perception, waits again.

Closing that gap is not one trick but a stack of them, and each leaves a fingerprint on the data you must collect. Latency gets filed under systems engineering, but the fixes reach back to how a demonstration has to be recorded in the first place.

The clock the robot cannot miss

Start with the numbers, roughly. A dexterous manipulation controller wants to update somewhere between 30 and 200 Hz, depending on the task and the hardware. Call it 50 Hz for a two-arm tabletop system: a new target every 20 milliseconds. A large VLA forward pass on an onboard GPU is far slower, commonly 50 to 200 milliseconds once you count the vision encoder, language backbone, and action decoder. Even at the optimistic end, the model makes one decision in the time the controller wanted five.

You cannot just run the controller slower. Drop a manipulation loop to 5 Hz and contact-rich motion falls apart: the gripper overshoots, force spikes go uncorrected, anything dynamic is hopeless. So the real question is never how to make one forward pass instant. It is how to keep a fast control loop fed by a slow model, neither one waiting on the other. Every technique below is an answer to that.

Action chunking buys time

The first and most important move is to stop asking the model for a single action. Instead the policy predicts a short horizon of future actions in one pass, a chunk, and the controller plays that chunk out step by step while the next inference runs. If one forward pass yields half a second of motion at 50 Hz, that is twenty-five commands from a single think. The model's latency is amortized across the whole chunk rather than blocking every step. This is why nearly every serious VLA, Physical Intelligence's π0 line among them, predicts chunks instead of steps.

A chunk is a bet on the future, though. The longer the horizon, the more the robot acts open-loop, executing a plan drawn from perception that is now old. Commit to too long a chunk and the policy stops reacting: it will finish a grasp on a cup that already moved. Too short and you are back to paying the latency toll every few steps. Chunk length is a direct trade between reactivity and compute, tuned per task.

A policy that thinks in bursts and acts open-loop between them can only be as good as data recorded fast enough, and timed precisely enough, to supervise every step it will take alone.

Asynchronous inference and the reflex layer

Chunking alone still has a seam. When one chunk ends, the next has to be ready, and if inference is slow the robot either pauses or lurches as a new plan overwrites the old one. Asynchronous inference hides the seam: the next chunk is computed in the background while the current one executes, and the two are blended at the boundary so the switch is continuous rather than a jump. Physical Intelligence has described exactly this kind of real-time chunking, generating the next action sequence before the current one runs out, so the robot never stalls waiting for the brain.

A second structural answer is to split the model by speed. NVIDIA's Isaac GR00T is built around a two-system design borrowed loosely from human cognition: a slower System 2, a vision-language model that reasons about the scene and the goal, and a faster System 1 that turns that intent into high-rate motion. The slow half runs at a few hertz; the fast half runs at the control rate. Google DeepMind pushed the same instinct to the edge with an on-device version of Gemini Robotics, a smaller model that runs locally so perception to action never round-trips to a server. The shape recurs across labs: let a heavy model set direction occasionally, and a light one handle the millisecond-to-millisecond reflexes.

Shrinking the model: distillation and quantization

The other half of the toolkit attacks the forward pass itself, and two techniques dominate.

Distillation trains a small student policy to imitate a large teacher VLA. The big model generates or scores behavior; the compact model learns to reproduce it at a fraction of the parameters and the latency. You trade away some of the teacher's generality for a policy that fits the robot's compute budget and hits the control rate. Quantization keeps the same network but shrinks its numbers, from 16-bit floats to 8-bit or even 4-bit integers, cutting memory traffic and speeding the matrix multiplies that dominate inference. Done carefully it costs little accuracy; done carelessly it erodes exactly the fine motor precision a manipulation policy needs most.

Neither is free, and both interact with everything above. A distilled reflex policy is easier to run asynchronously but plans a shorter horizon reliably. A quantized model is faster but noisier, which raises the bar on how clean its training data must be. Much of the published record for these tradeoffs lives in the robotics literature rather than on product pages, and it moves fast.

Latency is a data problem too

Now the through-line. Every one of these fixes assumes something about the data underneath it, and mostly the same thing: the demonstrations were recorded fast, densely, and with accurate timing.

Chunking is the clearest case. To train a policy to emit half a second of action at 50 Hz, the demonstration has to contain that half second at 50 Hz. Actions are supervised per step, so a chunk of twenty-five commands needs twenty-five ground-truth targets sampled at the rate the robot will run. Capture the same task at 10 Hz and there is no 50 Hz signal to learn: the fine correction between samples was never recorded, and no interpolation invents it. The control frequency you want at deployment sets a floor on the capture frequency you needed at collection.

Asynchronous blending adds a timing demand. If the data carries jittery or unknown latency between a camera frame and the action logged against it, the policy learns a smeared correspondence, and stitching chunks in real time amplifies the smear. Distillation and quantization add a quality demand: a smaller or lower-precision student has less capacity to average out noise, so it needs cleaner, more consistent demonstrations to reach the same reliability. Faster, tighter, better-timed data is not a luxury for real-time control. It is the precondition.

Latency, in the end, is not a footnote to the model. It is the discipline the physical world imposes on it. A VLA that dazzles offline is worthless if it cannot answer inside the window the task allows, and every method that earns it that window, from chunked horizons to distilled reflexes, reaches back to demand data recorded faster and timed tighter than the robot will ever move. The models will keep getting quicker. The harder, quieter question is whether the record they learned from was fast enough to begin with.