Why the robot data flywheel needs failures, not just wins
Success-only demonstrations flatter a robot policy and hide where it breaks. Why the data flywheel only compounds when you capture failures and near misses.
A robot policy that has only ever watched things go right has no idea what to do when they go wrong. Show it ten thousand clean grasps and it learns the clean grasp beautifully. Then, in deployment, the gripper clips the cup, the cup tips half a centimeter, and the policy is staring at a scene no demonstration ever contained. It has no move for this. The one skill it now needs, how to recover, was the one thing nobody thought to record.
Almost every capture effort is pointed at success. That is understandable: a demonstration is a thing you want the robot to imitate, so you keep the run where it worked and you throw away the fumble. But a policy improves fastest from the states where it fails, and those states are exactly what a success-only dataset leaves out. The data flywheel everyone wants, deploy, collect, retrain, only compounds if the collect step captures failures and near misses, mines them, and aims the next batch of capture at precisely those states.
The problem is structural, not a matter of discipline. Failures are rarer than successes in any curated set, they are messier to store, and they are much harder to label. Left alone, every dataset drifts toward survivorship bias: a tidy archive of easy wins that flatters the model and tells you almost nothing about where it breaks.
The policy fails where the expert never went
Start with the failure mode that has haunted imitation learning since the beginning. Train a policy by copying expert demonstrations and it learns the mapping from states to actions only for the states the expert actually visited. Those demonstrations are, by construction, a narrow ribbon of near-perfect trajectories. At deployment the policy makes a small error, that error nudges it off the ribbon, and now it sits in a state a little further from anything it saw in training, where its next action is a little worse. Errors compound. This is covariate shift, and it is why a policy with 99% per-step accuracy can still fail a long-horizon task most of the time.
The classic fix, DAgger and its descendants, is not a cleverer loss function. It is a data-collection rule: run the current policy, let it drift into the states it actually reaches, and gather expert corrections precisely there. Work out of Berkeley BAIR and the wider cs.RO literature keeps rediscovering the same point in new clothes: the data that repairs a policy is not more of the expert's ideal trajectory, it is coverage of the off-distribution states the policy visits when it starts to go wrong. Those states sit next to failures, because drifting toward a failure is exactly what they are.
Recovery is a skill, and it has to be shown
There is a difference between a policy that does not fail and a policy that recovers, and only the second one survives contact with the real world. Recovery is its own skill: re-grasping after a slip, re-approaching after a miss, backing out of a jam to try a different angle. None of it appears in a dataset where every episode starts from a clean reset and ends in success. If the model has never seen the transition from botched to fixed, it cannot produce that transition on demand.
This is why corrective, intervention-style collection matters. A human supervisor watches the policy run, takes over the instant it starts to fail, demonstrates the way out, then hands control back. The valuable frames are the ones bracketing the intervention: the slide into trouble and the climb back out. Getting them means letting the policy fail a little on purpose, which cuts against every instinct that says capture should be clean. A dataset that never lets the robot stumble can never teach it to catch itself.
Hard negatives, and the value of knowing what not to do
Successful demonstrations tell a model what to do. They are silent about what not to do, and that silence gets expensive the moment you need a model to judge, rank, or score behavior rather than just mimic it. A reward model, a value function, or a learned world model has to tell a good trajectory from a bad one. Feed it only good ones and it has no basis for the distinction. It has never seen the boundary it is supposed to draw.
Hard-negative mining is the discipline of finding the failures that sit closest to success, the near misses where the grip almost held or the peg almost seated, and weighting them heavily, because those are the cases that actually pin the boundary down. Google DeepMind and other groups building large behavior and world models increasingly treat deployment logs as a mine for exactly these states: the ones where the policy was uncertain, or where an autonomous run diverged from what a human would have done. An easy failure, the robot missing by a mile, teaches little. The hard negative, the failure that looked like a success until the last centimeter, is worth a hundred clean wins.
A dataset of only successes teaches a robot to be lucky, not robust. The states that decide a deployment are the ones nobody wanted to record.
Why the easy wins pile up and the failures do not
If failure data is so valuable, why is it so scarce? Because every part of the pipeline conspires to discard it. Teleoperation, still the workhorse of high-quality capture, naturally produces successes: a skilled operator rarely fumbles, and when they do, the reflex is to delete the take and redo it. Failures are rarer to begin with, and the good ones, the informative near misses, are rarer still. They are messier to store, often partial or ambiguous, with no clean end state to anchor them. And they are far harder to label. A success labels itself, the task is done. A failure raises questions a human has to answer: when exactly did it go wrong, what was the correct action at that moment, and what would recovery have looked like. That annotation is slow, subjective, and easy to get wrong.
The result shows up in the public corpora. The large teleoperation and cross-embodiment datasets the field trains on are overwhelmingly successful trajectories, because that is what was worth keeping under the old assumption that you imitate wins. Toyota Research Institute, in its work on large behavior models, has been candid that honest evaluation, and by extension honest training, depends on covering the cases where behavior degrades, not only the ones where it shines. You cannot even measure robustness without failures to test against, let alone train for it.
| Dimension | Success-only capture | Failure-aware capture |
|---|---|---|
| State coverage | The narrow ribbon of expert trajectories | Adds the off-distribution states a policy drifts into |
| Robustness at deployment | Brittle once errors compound | Holds up under drift and perturbation |
| Recovery behavior | Absent, no botched-to-fixed transitions | Explicit re-grasp and re-approach examples |
| Signal for value and reward models | No negatives, cannot draw the boundary | Hard negatives that pin the good/bad line |
| Collection difficulty | Low, successes come for free | High, you must let the policy fail on purpose |
| Labeling cost | Low, a success labels itself | High, someone must annotate when and why it broke |
The advantage belongs to whoever keeps the mistakes
The instinct to collect successes is not wrong, it is incomplete. A policy needs to see the shape of the task done well, and it needs to see the moments the task nearly came apart and was saved. The teams pulling ahead are not the ones with the largest pile of flawless demonstrations. They are the ones who stopped throwing the failures away, learned to label them, and started aiming the next capture batch straight at the states where their best model still breaks.