Real vs Simulation Data: The Honest Cost
Simulation data looks nearly free until the reality gap sends its invoice. An honest cost comparison of real vs simulation data for training robots.
A GPU farm can generate a million grasp attempts overnight, each one perfectly labeled, for roughly the cost of the electricity. Set that against a human collecting demonstrations one careful trajectory at a time, and simulation looks like it already won. The invoice just arrives later, written in a different currency: the reality gap.
Simulated data is not free. It is cheap to produce and expensive to trust. Real data is the reverse. Getting the accounting right, instead of quoting a marginal cost per sample and stopping there, is what separates a training strategy that ships from one that stalls in the sim-to-real gap.
My argument is simple. Even a program that leans hard on simulation still needs a real corpus to anchor and measure the gap, and pretending otherwise is how teams burn a year.
The seductive economics of simulation
The marginal cost of another simulated episode is close to zero. Ground-truth labels come free: pose, contact, and segmentation are all known to the simulator that drew the scene. Dangerous or rare situations can be generated safely and in bulk, and domain randomization widens coverage by scrambling textures, lighting, and physics parameters across runs. Work on scaling simulation and transfer from groups like the NVIDIA GEAR Lab research is built on exactly these strengths. On paper, the supply curve is nearly flat: want ten times more data, spend ten times the compute.
There is a reason this pitch lands in every board meeting. Compute scales in ways human labor does not. Double the cluster and you double the throughput, with no hiring, no scheduling, no fatigue. For locomotion and navigation, where the physics a robot cares about is mostly rigid-body dynamics that engines model well, the bet has paid off more than once. Quadrupeds have walked out of simulation onto real trails with very little real-world tuning. When the governing physics is clean, simulation is close to a free lunch.
The reality gap is a cost, not a footnote
Contact physics, friction, deformable materials, and sensor noise are exactly the things a simulator models worst. A policy that overfits to a simulator's quirks fails on real hardware, sometimes spectacularly. Closing the gap costs real engineering: system identification, better physics, and endless tuning of the randomization ranges. And you still need real data to check whether you closed it at all. The real-world manipulation work at Physical Intelligence, and real teleoperation datasets like DROID, are reminders that hardware is the final judge and it does not grade on a curve.
A concrete failure looks like this. A grasping policy trained in a simulator learns that a certain finger angle always works, because the simulated friction coefficient is one clean number. On real hardware, friction shifts with dust, wear, and humidity, and the same angle slips one time in five. The policy never saw that variance, so it never learned to correct for it. The bug is not in the code. It is in the assumption that a number the simulator made up describes the world.
| Dimension | Real data | Simulation data |
|---|---|---|
| Marginal cost per demo | High | Near zero |
| Label quality | Needs annotation, can be noisy | Perfect ground truth |
| Contact fidelity | Real by construction | Depends on the physics engine |
| Long-tail coverage | Limited by capture time | Expandable in software |
| Upfront cost | Capture rig and operators | Assets, scenes, physics tuning |
| Transfer risk | None, it is already real | The sim-to-real gap |
Where each actually wins
Simulation wins at locomotion and whole-body reinforcement learning, at massive parallelism, at scene variety, and at rare or dangerous events you would never stage for real. Real data wins at contact-rich manipulation, at deformable objects, at the true statistics of sensor noise, and at the small human strategies no reward function thought to specify. Aggregated real corpora like Open X-Embodiment are valuable precisely because they pool real behavior across many robots, not synthetic trajectories.
The split is not arbitrary. Simulation is strongest where the governing physics is well understood and cheap to compute, and weakest where it is not. Rigid-body locomotion sits on the strong side. A cloth fold, a poured liquid, a cable threaded through a clip, all of these sit on the weak side, where small modeling errors compound into behavior the real robot will not reproduce. Knowing which side of that line your task falls on is most of the decision.
The honest total cost
Simulation's hidden costs rarely make the slide. Someone has to build the 3D assets, author the scenes, tune the physics, and run the validation on real data that you cannot skip. Real data has its own hidden costs: the capture infrastructure, the operators, the cleaning and labeling. Add both columns up and the honest conclusion is that they are complements. Real data is not a line item you delete once the simulator works. It is the anchor that grounds the simulator and tells you whether its output means anything.
And the validation never really ends. Every time you change the simulator, retune the randomization, or target new hardware, you have to re-measure the gap against real data. That real evaluation set is not a one-time purchase. It is a standing cost, and it grows with how aggressively you lean on synthetic data in the first place. The more you simulate, the more real data you need just to keep trusting what the simulator tells you.
Simulation multiplies data you already trust. It cannot manufacture trust you never had.
How to split the budget
Treating simulated data as free is the most expensive mistake on the list. The right question is not real or simulated. It is which part of a skill can be safely multiplied in simulation, and which part must be grounded in real contact. Locomotion, navigation, and coarse scene variety are a bargain in sim. Contact-rich dexterity, deformables, and sensor noise still belong to real data. Put both on the same cost sheet and you stop seeing rivals, and start seeing two ends of one pipeline.