From demonstration to deployment: the robot training pipeline
Pretrain, post-train, sim2real, and evaluation are one loop closed by data and honest testing, not a checklist. A field guide to the robot training pipeline.
Watch a manipulation policy fail and you learn more than watching it succeed. A model that pours water flawlessly on the bench will miss the cup when the lighting shifts, when the table sits three centimeters higher, or when the mug is glossy instead of matte. The demo looked like intelligence. What actually broke was a pipeline, at a specific stage, for a reason you can name.
Most write-ups describe robot foundation models as a straight line: collect data, pretrain, fine-tune, deploy. That framing is convenient and wrong. Pretraining, post-training, sim2real, and evaluation are not a sequence you walk once. They are a loop, closed by evaluation, that you walk many times, and every trip changes what data you collect next. The teams shipping useful policies treat the four stages as one system with a feedback edge, not four departments.
This piece takes a position: the binding constraint in that loop is not model architecture, which is converging and increasingly shared, but the demonstration data and the honesty of your evaluation. Get those two right and mediocre architectures work. Get them wrong and the best VLA on the leaderboard still knocks over the cup.
Four stages, one loop
Before the details, here is the map. Each stage has a job, a characteristic data source, and a way it quietly fails.
| Stage | Goal | Where the data comes from | Dominant failure mode |
|---|---|---|---|
| Pretrain | General visuomotor priors | Pooled cross-embodiment and human video | Prior sits far from your robot and scene |
| Post-train | Task competence on your hardware | Teleoperated demos on the target robot | Too few demos, too correlated |
| Sim2real | Scale rare, costly, or unsafe conditions | Physics simulators plus domain randomization | Contact and deformables break the sim gap |
| Evaluation | Measure real progress, not proxies | On-robot trials, held-out scenes | Cheap metrics that do not predict field behavior |
Pretraining: breadth borrowed from other robots
No single lab can teleoperate enough hours to teach a model the visual and physical world from scratch, so pretraining borrows breadth. Open X-Embodiment pools roughly a million real trajectories from 22 robot types into one mixture, and models trained on the pooled set transfer better than models trained on any single robot. NVIDIA Isaac GR00T and Physical Intelligence's pi-zero both follow the same recipe: a vision-language backbone carries semantic priors, an action head learns motor priors, and both are pretrained on a heterogeneous mixture before they ever see your task.
Human video is the other half of breadth. Thousands of hours of everyday first-person recording (Ego4D) and roughly 1,300 hours of time-synchronized first and third person capture (Ego-Exo4D) give a model priors about how hands approach objects, how tasks decompose, and what a plausible next motion looks like. A person generates contact-rich, task-relevant behavior far faster and cheaper than a teleoperated arm. The catch: humans are not the robot, so this data helps as a prior and hurts as a direct target if you skip the next stage.
Post-training: the last mile decides the policy
Pretraining gives competence in general. Post-training gives competence on your robot, in your cell, on your task, and it is where most deployed performance is actually won or lost. Here the data is small and expensive: teleoperated demonstrations on the target hardware. DROID is the reference point for quality, 76,000 teleoperated trajectories across 564 scenes collected on a common setup, and it is valuable precisely because it is diverse and calibrated rather than merely large. Contact-rich corpora such as RH20T, on the order of 110,000 episodes, push the same idea toward forceful, multi-step tasks.
The failure mode is subtle. Two hundred demonstrations collected by one operator in one afternoon are highly correlated: same grasp approach, same lighting, same clutter. The policy overfits the operator's habits and looks excellent in evaluation that mirrors those habits. Open pipelines like Hugging Face LeRobot make it easy to standardize and share these datasets, which helps, but no format fixes a corpus that only ever saw one way of doing the task.
Sim2real: cheap data with a reality tax
Simulation is the only way to get millions of trials of a dangerous or rare condition without breaking hardware. Domain randomization, varying lighting, textures, masses, and friction so the real world looks like just another sample, is what lets a sim-trained policy survive contact with reality. This works remarkably well for locomotion and whole-body control, where the dominant physics is rigid-body dynamics that simulators model well, and where NVIDIA's GEAR Lab and the Isaac stack have pushed hard.
It works far less well for the tasks people most want: contact-rich manipulation of soft, deformable, or articulated objects. A simulator that cannot faithfully model a folding towel, a zipper, or a slipping grasp will train a confident policy that fails on the real thing. The honest rule of thumb: sim buys you dynamics and geometry cheaply, and it buys you contact and deformation at a steep and often hidden tax. Sim2real is not free data, it is a loan against your evaluation budget.
Evaluation: the bottleneck nobody demos
Every stage above is graded by the same judge, and the judge is expensive. Real robot evaluation is slow, noisy, and hard to reproduce: the same policy scores differently on Tuesday because the light changed. Simulation and offline metrics are cheap but routinely disagree with on-robot success, so a policy can climb an offline curve while getting worse in the cell. Toyota Research Institute's work on large behavior models is blunt about this: careful real-world evaluation across many trials and scenes is the part that actually tells you whether the last change helped.
Most robot demos show the best run out of many. The pipeline that matters is the one that tells you the average run, honestly, before you ship.
This is why evaluation, not architecture, closes the loop. A trustworthy eval tells you which stage to invest in next: broaden the pretrain mixture, collect more diverse post-train demos, or stop trusting a sim asset. A flattering eval sends you in circles.
The next policy you watch fold a shirt on stage tells you almost nothing on its own. The useful questions are which of the four stages it has actually closed the loop on, how many real trials stand behind the number, and whose demonstrations taught it the motion. Answer those and you can tell a pipeline from a highlight reel.