Imitation vs Reinforcement Learning for Manipulation
Why most robot manipulation stacks learn by imitation first and reach for reinforcement learning second, when each wins, and how the two combine.
Watch a modern manipulation policy learn to fold a shirt, and the surprising part is how little it sees. A teleoperator puts on a headset, drives the robot arms through the motion a few dozen times, and a policy trained on those clips starts folding on its own. Now ask a reinforcement learning agent to acquire the same skill from scratch on the same hardware. It needs a reward signal, thousands of attempts, and a person or a fixture to un-crumple the shirt after every failed try. Same task, very different bill.
That gap is the quiet reason most manipulation stacks in 2026 reach for imitation first and reinforcement learning second, if at all. It is not that reinforcement learning is weak. It is that imitation learning sidesteps the two things that make RL painful on physical robots: hand-designing a dense reward for a contact-rich task, and collecting millions of trials on hardware that wears out, drifts out of calibration, and cannot reset itself.
The useful question is not which method is better in the abstract. It is where each one earns its keep, and why the strongest systems today are built on imitation with reinforcement learning bolted on as a finishing step.
The reset problem, and why real-robot RL is expensive
Reinforcement learning needs three things that are cheap in a video game and costly on a robot: a reward function, a lot of interaction, and a way to start over. In a simulator you get all three for free. A humanoid can fall a hundred thousand times before lunch, and the scene snaps back to its initial state instantly. On real hardware, every fall is wear, every reset is a human walking over to stand the robot up or re-scatter the blocks, and every reward has to be measured by a sensor you actually have.
Reward design is the subtler trap. "Screw in the bolt" has no natural scalar. Teams end up shaping the reward term by term: distance to the target, alignment of the thread, a penalty for excessive force, and the policy cheerfully exploits every gap in that specification. Contact-rich tasks make it worse, because the informative reward often arrives only at the very end, after a long sequence of precise moves. Sparse reward plus long horizon is the exact regime where model-free RL is least sample efficient.
Simulation dodges the sample-cost problem but opens the sim-to-real gap: contact dynamics, friction, and deformable objects are precisely what simulators model worst. Massively parallel simulation and aggressive domain randomization have made locomotion a genuine RL success story, and platforms like NVIDIA's Isaac stack exist to push that pipeline. Dexterous, contact-heavy manipulation has been slower to cross the gap.
What imitation actually buys you
Imitation learning trades the reward function for demonstrations. The oldest form, behavioral cloning, just does supervised learning from observations to actions. Its classic weakness is compounding error: a small deviation carries the robot into states no demonstration covered, and the mistakes snowball. For years that made pure cloning fragile.
The recent gains come from fixing exactly that. Action chunking, popularized by ALOHA and its ACT model, predicts short sequences of actions instead of single steps, which smooths the drift. Diffusion policies, from the Toyota Research Institute and Columbia line of work, treat action generation as a denoising process and handle multimodal behavior, the fact that there are many valid ways to pick up a cup, without averaging them into mush. These methods learn usable skills from tens to a few hundred demonstrations per task, not millions.
Scale then raised the ceiling. Cross-embodiment datasets pooled demonstrations across labs: Open X-Embodiment gathered on the order of a million real trajectories spanning 22 robot types, and DROID added roughly 76,000 teleoperated trajectories collected in the wild across many scenes. Vision-language-action models trained on this kind of data, Physical Intelligence's pi-0, NVIDIA's GR00T, and DeepMind's Gemini Robotics among them, inherit semantic generalization from web pretraining and motor grounding from demonstrations. None of them learned to manipulate primarily by trial and error.
Reinforcement learning answers a question imitation cannot: what to do when there is no expert to copy. On real hardware, the cost of asking that question is the whole story.
Where reinforcement learning still wins
Imitation has a hard ceiling: a cloned policy is bounded by the people who demonstrated it. When a task requires behavior no human can teleoperate well, or precision beyond human patience, RL earns its cost.
- High-precision, contact-rich insertion. Peg-in-hole, connector mating, and in-hand reorientation involve forces and micro-adjustments that are hard to demonstrate but easy to score with a success detector. RL fine-tuning routinely closes the last few percent of success rate here.
- Dynamic and whole-body control. Locomotion, balance recovery, and throwing are the clearest RL wins, which is why locomotion teams at Boston Dynamics and Agility lean on it. These behaviors are physically consistent enough to train in simulation and transfer.
- Superhuman optimization. When you want the fastest, smoothest, or most force-efficient motion rather than a human-like one, an objective beats a demonstration.
The practical pattern is rarely pure RL. It is residual or fine-tuning RL: start from an imitation policy, then let RL correct it within a narrow envelope, using a learned or scripted success signal instead of a hand-shaped dense reward. You keep imitation's sample efficiency and safety, and spend expensive real-world interaction only where it moves the number.
Two methods, side by side
| Dimension | Imitation learning | Reinforcement learning |
|---|---|---|
| Core requirement | Expert demonstrations | Reward function and interaction |
| Data efficiency on hardware | Tens to hundreds of demos per skill | Thousands to millions of trials |
| Reset and safety burden | Low, demos are collected once | High, needs resets and guardrails |
| Can exceed the human teacher | No, bounded by demonstrators | Yes, it optimizes an objective |
| Hardest part | Collecting diverse, clean data | Reward shaping and sim-to-real |
| Sweet spot | Broad, semantic, multi-task skills | Precise, dynamic, contact-rich tuning |
The stack most teams actually ship
Read the recent releases and a common shape appears. A large policy is pretrained on demonstrations, teleoperated robot data plus, increasingly, human video, then adapted to a specific robot and task with a modest number of on-robot demos, and only then, if the last few percent matter, polished with reinforcement learning against a success detector. Toyota's Large Behavior Models and the VLA foundation models above all follow the imitation-first spine; RL is a scalpel, not the engine.
The reason is economic as much as technical. Demonstrations are a fixed, predictable cost that scales with people and rigs. Real-robot RL is a variable cost that scales with the thing you are least willing to spend: hardware hours and reset labor. When a demonstration is available, it is almost always the cheaper bit.
So the honest framing is not imitation versus reinforcement learning. It is imitation as the default that gets a competent policy cheaply, and reinforcement learning as the targeted tool that pushes past what humans can demonstrate. The teams shipping capable manipulation are not choosing a side. They are choosing an order: copy first, optimize second, and spend scarce real-world interaction only where copying runs out.