Teleoperation vs human video: two roads to robot data
Teleoperation gives perfectly aligned robot demonstrations at brutal cost; human video gives cheap scale you have to retarget. The tradeoffs, side by side.
Put a stopwatch on both. A skilled teleoperator in a VR rig might record a robot folding a towel a few dozen times in an hour, and every repetition burns a robot, an operator, and a controlled workspace. In that same hour, a head-mounted camera on a line cook captures hundreds of grasps, pours, and wipes for the price of a memory card. One stream is scarce and shaped exactly for training. The other is nearly free and slightly wrong. That mismatch is the whole story.
Both are answers to the same hunger. Imitation learning needs demonstrations, and the physical world hands out almost none for free, so two supply lines have grown up to feed it. Teleoperation puts a human in the loop of the robot itself, through leader-follower arms, VR controllers, or an exoskeleton, and records the robot's own sensors and commands as the human drives it. Human-video learning skips the robot entirely, records people doing ordinary tasks in the first person, then tries to turn that footage into something a policy can learn from.
The honest difference between them is not "quality" in the abstract. It is where the action label comes from, and who pays to line it up with the robot. Teleoperation buys that alignment up front, at a brutal unit cost. Human video buys scale up front and defers the alignment bill to the algorithm. This is a look at which invoice you would rather pay, and why the best robot programs are quietly refusing to choose.
Where the action label lives
Start with the one fact everything else hangs from. In teleoperation the robot is the agent, so every frame arrives paired with the exact joint commands and gripper state that produced the next frame, in the robot's own action space. Setting up supervised imitation is almost trivial: predict the recorded action. Human video hands you the opposite. You see intent and outcome in rich detail, but the "action" is a human hand with more than twenty degrees of freedom, moving under contact forces nobody measured. To use it you must reconstruct an action the robot can actually execute: estimate the hand pose, retarget it to a gripper, and infer the contact you never sensed. Roboticists call this the correspondence problem. Teleoperation solves it by construction. Human video makes it your problem, and every tradeoff below is downstream of that.
The teleoperation ledger
What teleoperation gives you is the cleanest signal in robot learning: data drawn from the exact machine you will deploy, with proprioception, timed action labels, and often force-torque and tactile channels that passive video can never recover. DROID collected on the order of 76,000 teleoperated trajectories, roughly 350 hours, across hundreds of scenes and many institutions, and it stands as a marker for what scaled teleoperation looks like. RH20T pushes into the contact-rich end, with more than 100,000 sequences carrying force-torque and audio alongside vision. And because no single lab can afford to start from zero, Open X-Embodiment pools teleoperated data across more than twenty robot types into a corpus on the order of a million trajectories.
- Action labels in the robot's own space, aligned frame by frame
- Proprioception and, on richer rigs, force-torque and tactile signals
- Timing that matches the control loop, not a viewer's moving camera
The bill comes due on throughput. One operator drives one robot for one hour and produces one robot-hour of data; there is no multiplier. Hardware wears, workspaces have to be supervised, and the scenes are whatever fits in the lab, so the long tail of real-world variation is exactly the part teleoperation is worst at covering. It is the highest-fidelity demonstration data in the field and the hardest to scale.
The human-video ledger
Human video wins on the two axes teleoperation cannot: scale and diversity. Ego4D gathered roughly 3,600 hours of first-person video of everyday activity across many countries and settings. Ego-Exo4D went further and captured skilled human activity from paired first- and third-person cameras, with expert commentary and skill annotations, well over a thousand hours of it, built specifically so machines can study how people do things well. This is an ocean next to teleoperation's bucket, and it grows for the cost of cameras.
The catch is everything teleoperation had for free. There is no robot, so no native action labels; no force sensing; a standing morphology gap between a five-fingered hand and a parallel-jaw gripper; and a viewpoint that drifts with a human head rather than a controlled wrist. So human video mostly enters training through the side door: as pretraining for vision-language representations, affordances, and world-model priors, and increasingly as retargeted pseudo-actions recovered from hand pose. NVIDIA's GEAR lab work on learning from human video and synthesizing neural trajectories is one active line here. The wager is that a mountain of cheap, wrong-embodiment data plus a little right-embodiment data beats either one alone.
Teleoperation pays for alignment and rations scale. Human video pays for scale and still owes you the alignment. Neither road is free; they just send the invoice to different places.
Two ledgers, side by side
| Dimension | Teleoperation | Human video |
|---|---|---|
| Who acts | The robot, driven by a human | A human, no robot present |
| Action labels | Native, in the robot's space | Absent; must be reconstructed |
| Embodiment match | Exact | Morphology gap to bridge |
| Force and proprioception | Usually captured | Not captured |
| Cost per usable hour | High; one robot-hour per operator-hour | Low; camera time only |
| Scale and diversity | Narrow, lab-bound | Broad, in the wild |
| Main role today | Fine-tuning and deployment data | Pretraining and priors |
| Chief risk | Cannot scale to the long tail | Correspondence and retargeting error |
Read it as two invoices, not a ranking. Each road buys one thing cheaply by paying dearly for the other.
The roads are converging
The most interesting engineering now lives between the two. Handheld gripper rigs, a real parallel-jaw gripper held in a human hand with a wrist camera, let a person collect "human video" that is already recorded in the robot's action space, collapsing much of the correspondence problem while keeping human-speed, in-the-wild collection. Exoskeleton and puppeted setups chase the same trick from the other side. Meanwhile the default recipe across the field has settled into a sequence: pretrain on human video for perception and priors, then fine-tune on teleoperation to lock in the exact embodiment. Cross-embodiment pooling, the instinct behind Open X-Embodiment, sits on top of both. The live question is no longer "which road" but "what mixture, and where do you spend your scarce teleop hours."
So do not ask which data is better. Ask what you are short of. Short of alignment, teleoperate, and pay in operator-hours. Short of scale, film humans, and pay in retargeting. Most teams are short of both, which is why the sharpest work has moved to the exchange rate between the two roads, and why the demonstration shortage will outlast every dataset released so far.