The sim-to-real gap, explained: where real data wins

A quadruped policy trained entirely in simulation can walk across a gravel slope it has never seen. A manipulation policy trained the same way will often fail to pick up a clear plastic cup, because the depth camera it depends on returns noise where the simulator rendered a clean surface. Same recipe, two outcomes. The difference is where each task sits on the sim-to-real gap.

The gap is not one thing. It is a bundle of mismatches between the world a simulator models and the world a robot acts in: contact dynamics it approximates, sensor noise it under-renders, friction and mass it guesses at, and a long tail of events it never generated. For some tasks those mismatches are small enough to paper over with randomization. For others, the mismatch is the task.

Simulation is not optional in modern robot learning, and this is not an argument against it. It is an argument for precision: knowing which parts of a robot's competence you can synthesize, and which parts you have to observe in the physical world. Get that wrong and you can spend a GPU cluster teaching a policy to be confident about a world that does not exist.

The gap is three gaps, not one

Lumping every failure into "sim-to-real" hides the structure. It helps to separate three distinct gaps, because each one closes differently.

The dynamics gap is the mismatch in physics. Simulators resolve contact with approximations, penalty forces or complementarity solvers, and model friction as an idealized Coulomb cone. A real tendon-driven hand adds backlash, cable stretch, and thermal drift that no default solver captures. The error is small per step and compounds over a long manipulation sequence.

The perception gap is the mismatch in sensing. Rendering is not the same as sensing. Real cameras bring rolling shutter, motion blur, and autoexposure; depth sensors return garbage on transparent, specular, or thin objects. A policy that learned from clean synthetic depth has no representation for the failure mode it will meet first.

The coverage gap is the mismatch in variety. A simulator contains only what someone modeled. The real world supplies the slightly crushed cereal box, the cable that snagged, the glare at 5pm through a west window. These are the unknown unknowns, and they are precisely the samples domain randomization cannot generate, because no one thought to parameterize them.

The three gaps do not close with the same tool. The dynamics gap yields to better solvers and system identification. The perception gap yields to better rendering and to real sensor data. The coverage gap yields to neither, because you cannot randomize over failure modes you have not imagined; only contact with the real world adds them to the training set.

Where simulation earns its keep

None of this makes simulation weak. For a large class of problems it is the right tool, and the results are not close. Legged locomotion is the clearest case: the relevant state is mostly proprioceptive, contact events are brief, and the reward is dense. You can run tens of thousands of parallel environments, reset for free, and randomize mass, friction, and latency until the policy stops caring about the exact numbers. Pipelines built around NVIDIA Isaac and GR00T lean on exactly this, and the sim-trained walking and whole-body controllers that ship on real hardware are evidence the approach transfers when the physics is well-modeled.

The pattern generalizes. Where dynamics are dominated by rigid bodies, where sensing is proprioceptive rather than visual, and where you can define a dense reward, simulation with domain randomization is hard to beat on cost per useful sample.

Where simulation quietly underfits

Move from walking to touching, and the ground shifts. Contact-rich manipulation is where simulators are weakest and where the consequences are least forgiving. Inserting a connector, folding a shirt, wiping a surface, threading a cable: each one depends on friction, compliance, and micro-slip that current solvers approximate poorly. Deformables make it worse, because cloth, food, and cable have effectively infinite degrees of freedom and no clean rigid-body prior.

A simulator is a hypothesis about physics. Contact is where the hypothesis gets tested, and it is where the hypothesis most often fails.

Perception has the same structure. Transparent and reflective objects, the exact things a home or lab is full of, are where synthetic depth diverges hardest from real sensors. And anything involving people, handovers, shared workspaces, reading intent, has no faithful simulator at all, because you are no longer modeling physics but human behavior. This is why the strongest generalist manipulation policies are trained on real interaction data. Physical Intelligence pretrains its policies on large, heterogeneous real robot datasets; Toyota Research Institute's Large Behavior Models are built on hundreds of hours of teleoperated demonstration; and Open X-Embodiment exists because pooling real robot data across labs beats any single simulator for coverage.

Long-horizon tasks compound the problem. A policy that is 99% reliable per step is only about 60% reliable over fifty steps, and simulator error is not zero-mean: it biases in consistent directions that a policy learns to exploit. What looks like mastery in the simulator can be an elaborate way of gaming a physics engine, and the bill comes due on real hardware.

How the gap is actually being closed

The useful teams do not pick a side. They compose. Three moves recur.

Calibrate the simulator to reality, sometimes called real-to-sim: use real trajectories to identify mass, friction, and latency, so the model you randomize around is centered on the truth rather than a guess. Randomize across the residual uncertainty, so the policy is robust to the parameters you could not pin down. And co-train on real demonstrations, so the policy is anchored to real sensor statistics and to the messy tail that no simulator produced. Modern vision-language-action policies follow this shape: pretrain broadly, often on real and even human video, then fine-tune on the target embodiment.

That last ingredient is why real datasets have become infrastructure. DROID spread manipulation collection across 564 scenes and 13 institutions; Ego-Exo4D captures paired first-person and third-person human activity that gives policies a prior on how tasks are actually performed. Simulation multiplies the data you already understand. Real capture originates the parts of the distribution you do not.

The gap, in the end, is less a bug to eliminate than a map to read. Spend simulation where physics is well-modeled and cheap to randomize. Spend real data where contact, deformation, and human context live. Teams that treat the two as complements, not substitutes, are the ones whose robots work on the first real try more often than not.