Teaching robots from internet video: promise and limits

Every minute, the internet gains thousands of hours of fresh video, and that is one estimate across one slice of it. Somewhere in the pile, right now, someone is filming themselves chopping an onion, changing a tire, folding a fitted sheet, coaxing a stuck jar lid open with a dish towel. Billions of hours already sit there: a running record of human hands doing very nearly every task a household robot will ever be asked to perform. To anyone trying to train a robot, that looks like the jackpot, the text-scale corpus manipulation never had.

The pull toward that corpus is really an analogy to language. Large language models got good because humanity had already written the internet, and every page was a free training example produced for some other reason entirely. Robot manipulation was born without that inheritance: nobody instruments their kitchen while they cook. Web video looks like the closest substitute anyone will find. Pretrain on the whole churning mess, the reasoning goes, and a policy walks in having watched a lifetime of people handling the physical world.

It half works, and the half matters. Web video is one of the richest priors in robotics, and every serious humanoid program now leans on it. But there is a wall running through the middle of the jackpot, and it is not a wall of quantity. A clip of a hand doing a task is missing the one thing a robot most needs to copy: what the hand actually did. This is a piece about what the internet gives a robot for free, and the ingredient no amount of scraping will ever contain.

The free lunch is real

Start with the good news, because it is substantial. A model pretrained on internet video learns things that are genuinely hard to get any other way.

Visual and object priors. After enough footage, a network knows what a drawer, a bottle cap, a zipper, a whisk look like, from every angle and under every light. It carries an internal picture of the objects a robot will meet, before the robot meets one.

Semantic and activity priors. The internet has shown the model pour and fold and wipe tens of thousands of times, along with the order the steps come in and what a finished task looks like. That is a map of human activity a policy could never build from a few hundred lab demonstrations.

Coarse dynamics. Water falls, a dropped ball bounces, a soft bag slumps, a rigid box does not. Watched at scale, these regularities become a physics-shaped prior, which is exactly the raw material the world models now central to the field are built from. NVIDIA's GEAR Lab leans on human and internet video, plus synthetic trajectories, to bootstrap humanoid skills in its Isaac GR00T stack, precisely to bank these priors before a single robot demonstration is spent.

Human affordances. How people actually grip a mug by the handle, brace a box against the hip, steady a cutting board with the off hand. Those habits are dense in web video and thin nearly everywhere else.

The wall: the internet shows the outcome, not the action

Now the deficit, and it is structural rather than fixable with more hours. Web video is observational. It records what a scene looked like, never what the person commanded their body to do. Four things go missing, and a robot needs all four.

The action label is gone. A controller learns by regressing to a target: given this observation, output that motion. Web video hands you the observation and withholds the target. The joint trajectory, the finger closure, the timing tied to a control loop, none of it was recorded, because the camera was pointed at the result, not wired to the mover.

Force and proprioception are gone. The three newtons of grip that kept the glass from slipping, the weight of a full kettle, the moment contact begins, the tactile report from the fingertips, all of it is invisible to a lens. You cannot infer from pixels a quantity the pixels never encoded.

The hands themselves are only a guess. You can estimate a hand pose from video, but it is an estimate: noisy, scale-ambiguous, and occluded exactly when it matters most, in the final centimeters where the fingers wrap the object.

Then the problem specific to the internet, as opposed to deliberately captured human video. The viewpoint is wrong and unstable. Most web footage is third-person, shot by a bystander or a tripod, then edited, cut, zoomed, and color-graded, with a camera whose position and intrinsics you do not have and which jumps between shots for reasons that have nothing to do with the task. A robot will perceive the world from cameras bolted to its own head and wrists. The internet almost never shows that view. So even the pixels arrive from the wrong place, moving to a rhythm set by an editor rather than by the work.

The internet records what a task looked like from across the room. A robot has to reproduce what the hands did from inside the work. Those are not the same footage, and the gap between them is the entire problem.

What the corpus gives, and what it withholds

Read the three columns as one sentence: the internet donates the left column, can never yield the middle one, and the right column is the bill someone has to pay on purpose.

The playbook everyone converged on

Faced with that split, the field settled on one recipe with remarkable speed: pretrain broad, then align narrow. Soak up the ocean of web video for the visual, semantic, and dynamic prior, then finetune on a small, precisely captured set of demonstrations that carry the action, force, and viewpoint the video could not. The web half is nearly free and getting cheaper every quarter. The aligning half is scarce, expensive, and decisive.

The strongest systems all run some version of it. GR00T bootstraps on human video and synthetic motion, then grounds on real robot data. Physical Intelligence trains cross-embodiment policies on a broad base and aligns them to specific hardware with targeted demonstrations. Google DeepMind folds a video-trained prior into Gemini Robotics and finetunes for control. The web corpus is the same commodity for all of them; what differs is the quality of the small aligning set each can bring to bear.

Why the small set decides everything

Here is the counterintuitive part. The scarce, load-bearing ingredient is not the web video. That is abundant, cheap, and shared by everyone with a network connection. The scarce ingredient is the aligning demonstration: multimodal, first-person, action-labeled, force-annotated, recorded to a protocol. That data does not scrape, and it does not fall out of the internet, because almost nobody was wearing the rig while they did the task.

And it is the aligning set, not the web pretraining, that sets the ceiling. Another ten thousand hours of ambient footage nudges the prior a little. A cleaner, richer aligning set, one that actually captured the forces and the first-person view, moves the policy a lot. Deliberately captured corpora exist for this reason, and they are small for the same one: Ego4D assembled thousands of hours of first-person human activity, instrumented and recorded on purpose, and even that was built largely for perception rather than for driving a gripper. The record a controller truly wants is rarer still.

The internet will stay the cheapest teacher a robot has ever had, and it will keep getting better at showing a machine what the world looks like and how it tends to move. What it cannot do, ever, is show a robot what its own hands were supposed to do, because that was never in the frame. That footage has to be made, on purpose, one captured task at a time. The jackpot was always only half a jackpot, and the valuable half is the one still waiting to be recorded.