Synthetic Robot Data vs Real: The Mix That Works

NVIDIA's robotics group recently reported generating, on the order of, 780,000 synthetic manipulation trajectories in about 11 hours with a generative pipeline built around its Isaac GR00T models. For contrast, the DROID dataset, 76,000 real teleoperated trajectories, took a 13-institution consortium roughly a year to collect. If trajectory count alone decided the matter, the argument would be over.

It is not over. Policies trained on only-simulated or only-generated data keep hitting the same wall: the robot looks fluent in the demo reel, then fumbles the first real towel, the first unlit drawer, the first object whose friction the simulator guessed wrong. The useful finding of the last two years is not that synthetic data works, nor that it fails. It is that the recipes that generalize are almost always blends, and the blend ratio is a design choice with consequences.

So the question worth asking is not "synthetic or real." It is sharper: which parts of your data distribution can you cheaply fake, and which parts must you earn in the physical world? Moving from a side to a ratio is the whole point.

Four kinds of data, four different jobs

Robot learning pulls from four supply lines, and each is good at something the others are bad at.

Real robot teleoperation gives you exact, on-embodiment action labels: the joint commands that actually produced the motion. It is also the most expensive line to run. DROID, Open X-Embodiment (more than a million real trajectories pooled from 60-plus datasets across 22 robot types), and RH20T (over 100,000 contact-rich sequences) exist because collecting this data at scale is genuinely hard.

Human egocentric video is the opposite trade. Ego4D alone is roughly 3,670 hours of first-person human activity, and Ego-Exo4D adds synchronized third-person views. The coverage of real hands doing real tasks is enormous and the marginal cost is low, but there are no robot action labels and a real embodiment gap between a human hand and a two-finger gripper.

Physics simulation (Isaac Sim, domain randomization, and friends) is cheap to scale, gives perfect labels, and lets you rehearse dangerous states safely. Its weakness is the sim-to-real gap, which is worst exactly where manipulation lives: contact, friction, and deformable objects.

Generated or neural video, the newest line, uses world models to synthesize trajectories, as in GR00T-style neural data generation. It is cheap and visually diverse, but the physics can be quietly hallucinated and the action labels are approximate.

Four sources of robot training data and what each is actually good for

Source	Action labels	Marginal cost	Edge-case coverage	Main failure mode
Real teleoperation	Exact, on-embodiment	High	Narrow	Overfits to one rig and lab
Human egocentric video	None (embodiment gap)	Low	Very broad	Human-to-robot morphology mismatch
Physics simulation	Exact	Low	Tunable	Sim-to-real gap on contact and cloth
Generated / neural video	Approximate	Low	Broad (visual)	Hallucinated physics

The mixes that actually ship

Look at what the leading systems disclose about their training data and the pattern is consistent: nobody uses one source.

NVIDIA describes Isaac GR00T as trained on a mixture of captured real data, human video, and synthetically generated trajectories, precisely so the model is not hostage to any single line. Physical Intelligence has been explicit that its pi0 and pi0.5 models co-train on a heterogeneous, cross-embodiment mixture, with pi0.5 framed around open-world generalization from broad and varied data rather than one clean corpus. Open X-Embodiment showed that co-training a single policy on dozens of pooled real datasets beat training on any one of them, which is the co-training argument in miniature. And Toyota Research Institute's Large Behavior Models lean on large volumes of real teleoperation augmented, not replaced, by other sources.

The takeaway is structural, not incidental. When a team can afford exactly one source, it picks real teleoperation and starves on coverage. When it can afford many, it keeps a real anchor and buys breadth from the cheaper lines. The mixture is the strategy.

How to reason about the ratio

If the answer is a mixture, the practical skill is choosing proportions. A few heuristics hold up across teams:

• Spend real, on-robot data where contact and reward-relevant dynamics live. Grasping a deformable, judging slip, applying force: these are the behaviors a simulator or a video model is most likely to get subtly wrong, so pay for the real thing here.
• Use simulation for coverage and safety. Rare failures, dangerous states, and pure geometric variety are cheap to manufacture and tedious to collect physically.
• Use generated video for visual and semantic diversity. Lighting, clutter, and backgrounds are exactly what neural generation varies well and what teleoperation rigs sample poorly.
• Anchor the whole thing with a thin slice of real in-domain data. A small, well-chosen fine-tuning set on the target robot often closes most of the gap left by simulated or generated pretraining.

The proportions shift by problem. In locomotion reinforcement learning it is common for the overwhelming majority of experience to be simulated with only a light real-world fine-tune. In contact-rich manipulation the balance tilts back toward real teleoperation, because that is where the simulator's errors bite hardest.

The useful question is not whether synthetic data works. It is which slice of your distribution you are willing to fake, and which slice you have to earn in the physical world.

Where the ratio breaks

Blends fail in specific, diagnosable ways. The classic is the sim-to-real gap: a policy that mastered a rigid-body simulation meets a real cloth or a compliant object and the learned dynamics no longer hold. Generated video introduces a subtler version, hallucinated physics, where a frame sequence looks plausible but is dynamically impossible, and a policy that trains on it learns the wrong causal structure. Too much clean simulation creates covariate shift: the robot never saw a near-miss, so it has no recovery behavior when the real world drifts off the demonstrated path. And there is a provenance failure that has nothing to do with dynamics, mixing datasets whose licensing and collection consent are unclear, which is a growing liability as regulation such as the EU Data Act sharpens.

None of these are exotic, and each has a standard mitigation: more domain randomization, better real anchors, explicit recovery demonstrations, a clean provenance trail. But every mitigation costs something, which is why the ratio behaves like a budget, not a formula.

The teams making real progress are not picking a side in a synthetic-versus-real debate that was never the right frame. They are tuning a mixture, source by source and task by task, and treating the ratio itself as a claim about the world, one worth measuring rather than guessing.