Why world models entered the humanoid race

In 2024, Google DeepMind trained a model called Genie on tens of thousands of hours of internet video of people playing 2D platform games. Nobody labeled a single frame with what the jump button does. Yet you can hand the trained model one still image and play it: press left and the character walks left, the scene scrolls, gravity pulls it back down. There is no game engine underneath, only a network that learned how pixels tend to follow the actions of whoever held the controller.

That trick has a serious name and a serious purpose. A world model is a learned predictor of how a scene will evolve, optionally conditioned on an action. The humanoid field wants it badly for one blunt reason: real robot demonstrations are slow and expensive, and video is neither. Open X-Embodiment pooled on the order of a million real-robot trajectories across 22 different machines, and that is still a rounding error next to the billions of hours of human activity already sitting on the internet. If you can learn physics-shaped priors from video without a robot in the capture loop, you route around the bottleneck.

That is the promise, and it is why world models moved from a research curiosity to a line item on nearly every humanoid roadmap in about eighteen months. The catch is quieter. A model that predicts what the next frame looks like is not the same as a model that knows what it would feel like to cause that frame. Passive video omits precisely the signals a controller runs on. This is a piece about why the pivot happened, and where learning from video runs out of road.

What a world model actually is

The term covers at least three different machines, and conflating them causes most of the confusion.

The first is a learned simulator. In the Dreamer lineage of model-based reinforcement learning, an agent trains a compact model of its environment and then plans by rolling that model out in imagination, cheaper and faster than touching the real world. The second is representation pretraining: predict future frames or future embeddings, in the style of joint-embedding predictive architectures, and keep the features that fall out as a visual backbone for control. The third is an action-conditioned video predictor, a neural simulator that generates a plausible next few seconds given a candidate action, then uses those futures to score plans or to synthesize training data. NVIDIA's Cosmos world foundation models, wired into the GEAR Lab's GR00T stack, sit squarely in this third bucket.

The distinction that matters most is passive versus action-conditioned. A passive model answers what happens next. An action-conditioned model answers what happens next if I do a specific thing. Only the second is directly useful for control, and the second is far harder to train, because the internet is full of the first kind of video and almost none of the second.

Why the humanoid labs pivoted to video

Follow the economics. A teleoperated robot trajectory costs an operator, a machine, and wall-clock time. Human video costs a download. The gap is several orders of magnitude, and it points in only one direction.

So the moves came quickly. NVIDIA released Cosmos as an openly available world-model platform aimed at exactly this pretraining problem. 1X Technologies ran a public world model challenge, asking researchers to predict future frames of its home robot from past ones. Toyota Research Institute leaned on broad pretraining for its large behavior models, and Google DeepMind folded a video-trained prior into the reasoning of Gemini Robotics. Meta's V-JEPA pushed the representation route, predicting in embedding space rather than pixels. Different bets, one shared thesis: let video supply a prior about how objects move and how people handle them, then spend scarce robot data only to align that prior to a specific body.

What video teaches, and what it quietly omits

Passive video is genuinely rich. It carries appearance dynamics, object and scene motion, and a huge distribution of human affordances: how people actually grip a mug, brace a box, or steady a cutting board with the off hand. Those are exactly the priors a manipulation policy struggles to collect on its own.

What video does not carry is the other half of control. It has no record of the command that produced the motion, no contact forces at the fingertips, no proprioception, no tactile signal, and often no reliable 3D. Worse, a video model is trained to look right, not to be right. It optimizes visual plausibility, so it will cheerfully render a mug that slides without friction or a finger that clips through a handle, because those frames are perfectly believable. Physical correctness is not in its loss function.

The action label is the expensive part

The real gap is not resolution or hours, it is the action. Passive video is observational data: it shows what happened. Control needs interventional data: what happens if I push here rather than there. That difference is the old wall between correlation and causation, and no amount of extra YouTube fixes it, because the counterfactual was never recorded.

The datasets that do close it are small for exactly that reason. Open X-Embodiment, DROID, and RH20T are valuable because a human sat in the loop and the action was logged alongside the pixels. Ego-Exo4D goes further, pairing a first-person view with synchronized third-person cameras and dense step and action annotations, so the same motion is legible from the mover's eyes and from outside. Every one of these took deliberate capture, which is precisely why there are thousands of hours of them and not billions.

There is a second, subtler tax. Roll a learned model forward over a long horizon and small per-step prediction errors compound. Without grounded feedback to correct the drift, an imagined rollout that looks fine for a second becomes nonsense by the tenth. A prior can start a policy. It cannot, by itself, keep one honest.

A world model trained on video learns what the next frame looks like. Control needs to know what the next frame would feel like to cause, and that signal was never in the pixels.

How the two halves fit together

Stated as video versus robot data, the debate is a false choice. The working pattern across the strongest systems is a stack: pretrain broad, on video and learned representations, for the prior, then finetune narrow, on action-conditioned and contact-rich demonstrations, for the last mile. Toyota's large behavior models, Physical Intelligence's pi-zero, and Gemini Robotics all run some version of this two-stage recipe. The video half is getting cheaper every quarter. The scarce, decisive ingredient is the other half: action-conditioned, force-annotated, first-person demonstration that tells the model not just what the world did, but what a body did to it.

World models are the right answer to a real scarcity, and they will keep getting better at predicting pixels. But the humanoid that actually ships will be the one whose world model was corrected by data that knew what the action was. Video shows a robot the world. Something still has to show it the difference between watching and doing.