Post-training a VLA: from generalist to specialist policy
Fine-tuning turns a generalist robot policy into one you can deploy. A concrete look at VLA post-training: methods, data, failure modes, and evaluation.
A folding policy that hits 90% success in the lab can collapse to a coin flip the moment someone swaps the towel for a stiffer one, slides the table six inches, or lets the afternoon sun into the room. The weights did not change. The world did. That gap, between a policy that demos and a policy that ships, is almost always closed in post-training, not in the pretraining run everyone talks about.
A vision-language-action (VLA) model maps camera frames plus a language instruction straight to robot actions. A generalist checkpoint, NVIDIA's Isaac GR00T N1 or Physical Intelligence's π0, has already absorbed a wide slice of embodiments, scenes, and verbs. It carries a rough prior for what "pick up the cup" means and how an arm sweeps toward a target. What it has never seen is your gripper, your camera extrinsics, your lighting, or your definition of done. Post-training is the step that turns that broad prior into a policy that clears one job reliably.
The division of labor is uncomfortable: pretraining sets the ceiling, post-training decides whether you reach it. Teams burn months choosing a base model, then lose the deployment on a fine-tuning recipe they treated as an afterthought. What follows is what that recipe actually contains, where it breaks, and why the composition of the fine-tuning data matters more than its raw volume.
What the generalist prior actually buys you
Pretraining on pooled, cross-embodiment data buys generalization you cannot get from task data alone. The Open X-Embodiment effort showed it plainly: pooling on the order of a million trajectories across 22 robot types and co-training one model beat the same architecture trained on any single robot's data. Representations transfer. An arm that has seen thousands of kitchens brings a usable notion of "cup" and "handle" into a kitchen it has never entered.
What the prior does not buy is competence on the exact distribution you care about. Toyota Research Institute's work on Large Behavior Models sharpens the point: a pretrained policy does not merely start higher, it needs fewer demonstrations to reach a target success rate on a new skill. Pretraining shifts the whole data-efficiency curve down and to the left. Post-training is how you spend that efficiency.
A generalist VLA hands you a policy that is plausible everywhere and reliable nowhere. Post-training is the negotiation over where you spend the reliability.
The post-training recipe, and what each knob does
"Fine-tune the VLA" hides at least four different operations. Picking the wrong one costs weeks.
| Approach | What changes | Typical data | Main failure mode |
|---|---|---|---|
| Full fine-tuning | All weights | Hundreds to a few thousand target demos | Catastrophic forgetting, overfit to the rig |
| Parameter-efficient (LoRA, adapters) | Small added weights | Tens to hundreds of demos | Too little capacity for a large domain shift |
| Co-training | All weights, mixed batches | Target demos plus a slice of the pretraining mix | Needs the pretrain data, plus a mix ratio to tune |
| RL post-training | Policy under a reward | On-robot or simulated rollouts | Reward design, sample cost, safety |
Most shipped systems combine them. Physical Intelligence has described π0 as a pretrained base that is then post-trained on curated, high-quality data for the target behavior, and co-training against a slice of the original mix is the standard defense against forgetting. Google DeepMind's Gemini Robotics is presented the same way: a broad model that is specialized to new tasks and embodiments, including a smaller on-device variant that teams can adapt with a modest set of demonstrations.
How much data, and of what kind
The volume question has a boring answer and an interesting one. The boring answer: for a single well-scoped manipulation task on a fixed rig, a few hundred clean teleoperated demonstrations often move a pretrained VLA from unreliable to usable. The interesting answer is that composition beats count. A hundred demonstrations that cover the awkward cases, the missed grasp, the object at the edge of the workspace, the recovery after a slip, teach more than a thousand near-identical successes.
Provenance and consistency matter as much as the raw trajectories. DROID is instructive: roughly 76,000 teleoperated trajectories gathered across hundreds of scenes and dozens of buildings, deliberately built for visual and spatial diversity rather than sheer size. Fine-tuning data inherits the biases of how it was collected. If every demo was recorded under the same lamp with the same three objects, the policy will quietly learn the lamp. Diversity in the fine-tuning set is not a nice-to-have, it is the thing that survives contact with a new room.
Where fine-tuning quietly breaks
Three failure modes recur. The first is catastrophic forgetting: push all the weights hard toward one task and the model loses the language grounding and general dexterity that made the base worth using. Co-training and parameter-efficient methods both exist mostly to fight this.
The second is spurious correlation. A policy fine-tuned on a narrow set learns whatever predicts reward in that set, including the tablecloth, the time of day, or the operator's habit of always starting from the same pose. It looks brilliant in evaluation and falls apart when one of those incidental cues moves. The third is action-space mismatch: a base model pretrained across many embodiments encodes actions in one convention, and if your robot's control interface, frequency, or gripper geometry disagrees, the fine-tune spends its budget just re-learning the interface instead of the task.
Evaluation is the part teams underweight
A fine-tuned policy cannot be trusted on training-set loss. Real-robot evaluation is expensive, slow, and statistically noisy, which is exactly why it gets skipped. Toyota Research Institute has been unusually public about running blinded, statistically controlled hardware evaluations with enough trials to separate a real gain from luck, and the discipline is worth copying. A policy that reports 8 of 10 successes over 10 trials and a policy that reports 80 of 100 are not the same claim. Decide up front how many trials, under how much variation, count as "done," and hold the fine-tune to it.
The generalist checkpoint is the easy 80 percent. The specialist you can actually deploy lives in the last stretch: a few hundred demonstrations chosen well, a fine-tuning method matched to the size of the domain shift, and an evaluation honest enough to tell you when to stop. Treat that stretch as the main event, because for anyone shipping a robot, it is.