Why human demonstration data scales better than scripted data

A teleoperator sits behind a humanoid for six hours and collects four hundred clean demonstrations of a single task: pull the drawer, set down the mug, close the drawer. Every trajectory is smooth, labeled, and nearly identical. Down the corridor, a cafe's worth of people open drawers all morning: with a hip, with two fingers, with a full fist, while holding a phone, again after the drawer sticks. One of those two streams looks like training data. The other looks like the world.

That contrast is the whole argument. Scripted data optimizes for cleanliness and control. Human demonstration optimizes for coverage. At the scale where robot policies start to generalize, coverage is the property that pays, and the public datasets now make the case in numbers rather than vibes.

Two definitions before we go further. By scripted data I mean trajectories a machine or a tightly constrained teleoperation protocol produces on purpose: motion-planned pick-and-place, simulated rollouts with domain randomization, the same task repeated until the buffer is full. By human demonstration data I mean recordings of people actually performing tasks, increasingly captured from a first-person, egocentric point of view. The difference is not quality per frame. It is the shape of the distribution.

What scripted data actually buys you

Scripted and simulated data has real virtues, and it is worth being honest about them. It is cheap to mass-produce once the pipeline exists. It is perfectly labeled: you know the action, the contact, and the object pose, because you generated them. It is safe, repeatable, and easy to balance across classes. For pre-training a policy's low-level motor priors, this is a strong floor.

The failure mode is structural. Scripted data can only cover the variation you thought to encode. Domain randomization widens the distribution, but only along the axes you chose to randomize: lighting, textures, friction, mass. The cases that break deployed robots tend to sit off those axes, in the long tail of cluttered shelves, deformable objects, half-open containers, and human interference. A model trained mostly on the script learns the script. It performs the task the way you demonstrated it, then stalls the first time the world declines to match.

Coverage is the whole game

Generalization has a plain operational meaning: performing well on states you did not train on. To get there you need training states close to your test states, which is to say you need coverage. More of the same narrow task does not add coverage; it adds confidence on ground you already hold.

Open X-Embodiment put a number on this. Pooling roughly one million real trajectories across 22 robot embodiments and 60 constituent datasets, models co-trained on the mixture outperformed the same models trained on any single source, and skills transferred across robots that had never shared a dataset. The driver was not the volume of one narrow source. It was diversity. DROID makes the environmental version of the point: 76,000 teleoperated trajectories collected deliberately in the wild across 564 scenes and 52 buildings, with policies co-trained on it generalizing to unseen scenes markedly better than in-lab data of similar size.

The practical reading is a curve. Adding your ten-thousandth drawer-pull in one kitchen bends toward zero return. Adding a new kitchen, a new object, or a new strategy keeps paying. Whatever data source bends that curve most slowly is the one that scales.

A policy trained only on scripted trajectories knows one correct way to open a drawer. A policy trained on human demonstrations has seen a thousand almost-wrong-but-recovered ones, and recovery is most of what robustness is.

Why humans are the better sampler

Here is the leverage. People already perform the long tail every day, at no cost to any data budget, and egocentric capture turns that behavior into a corpus. Ego-Exo4D recorded roughly 1,300 hours of skilled activity from 740 participants across 13 cities, with time-synchronized first-person and third-person views; the earlier Ego4D effort gathered on the order of 3,700 hours from hundreds of camera wearers in dozens of locations. No robot-teleoperation program reaches that spread of people, places, and strategies, because it cannot: every hour costs a robot and an operator.

The field's own recipes already concede the point. NVIDIA's Isaac GR00T foundation model trains on a mixture that leans on human video alongside real-robot and synthetic trajectories, and Physical Intelligence trains its cross-embodiment policies across many robots and tasks rather than one clean source. When the groups building humanoid foundation models reach for diverse human and cross-embodiment data, that is not decoration. It is load-bearing.

The tradeoffs, side by side

None of this makes human data strictly superior. It is a different point on a set of tradeoffs, and choosing well means seeing the whole board.

Where human-first breaks, and the honest fixes

The column that should worry you is action labels. A human demonstration shows you the hand, not the joint commands; you observe an outcome, not a controller. Turning it into something a robot can imitate takes hand-pose and object-pose estimation followed by retargeting onto the robot's morphology, and every one of those stages compounds error. A five-fingered regrasp has no clean image on a two-finger gripper, so some human strategies simply do not transfer.

The fixes are known and increasingly standard. Paired capture is the main one: DROID-style multi-view rigs, Ego-Exo4D's synchronized ego and exo cameras, and RH20T's paired human-and-robot sequences all exist to give the retargeting step a supervised bridge. The correct conclusion is not that human data replaces robot data. It is a division of labor: spend abundant human demonstration on the diverse base, and spend scarce robot-hours on grounding and the last-mile skills where the embodiment gap actually bites.

The base and the tip

Treat scripted data for what it is: a clean, cheap floor for motor priors, not a path to a general policy. Coverage is the property that turns a policy from a demonstration into a skill, and human demonstration is the widest, cheapest source of it we have. The teams that scale will look less like a fleet of robots grinding one task and more like a wide human base, captured well, with robot time spent where it is worth the most. Collect the world, then teach the body.