Data Quality vs Quantity in Robot Learning

A manipulation policy folds a blue towel in the lab at a success rate that looks like a finished product. Swap the towel for a red one, push the table into afternoon sun, and the same policy stalls, reaching for a crease that is no longer where it expects. The team's first instinct is the obvious one: collect more towel-folding demonstrations. It rarely helps as much as the hours suggest.

This is the central puzzle of robot learning right now. Real robot data is expensive, teleoperated data especially, and the field spent three years assuming the fix was simply more of it. Then the Toyota Research Institute, building its Large Behavior Models, kept hitting a different result. The demonstrations that moved the needle were not extra repetitions of a task the model already handled. They were demonstrations that covered conditions the model had never seen. Coverage, not raw hours.

The distinction sounds academic until you price it. A thousand near-identical demonstrations and a thousand demonstrations spanning new objects, lighting, viewpoints, and recovery behaviors cost about the same to collect, yet they buy very different models. This piece unpacks why, what the diversity findings actually say, and how to read a dataset for coverage instead of for its headline hour count.

The lab-to-kitchen gap is a coverage problem

Most manipulation policies today are trained by imitation: a human teleoperates, the robot learns to copy. Imitation learning has a well-known failure mode. The policy is only reliable on the distribution of states it saw during training, and the moment the robot drifts into a state no demonstration covered, its errors compound. A small misgrasp nudges the scene into unfamiliar territory, the next action is slightly worse, and within a few steps the trajectory has left the training distribution entirely.

More demonstrations of a task the model already solves do little for this. They deepen a peak the policy has already found, sharpening behavior on states it can already handle, without widening the region of states it can recover from. What actually helps is support: demonstrations that touch the parts of the state space where the policy currently has nothing to imitate. That is why a red towel under new light can break a policy that folds blue towels perfectly. The failure is not a folding failure. It is a coverage gap.

What the diversity findings actually say

TRI's Large Behavior Model work sharpened a point the Open X-Embodiment collaboration had already hinted at: pooling diverse data across many tasks, and even many robot bodies, tends to improve performance on any single task rather than dilute it. The RT-X models trained on the pooled Open X-Embodiment data showed positive transfer, where a policy trained on many embodiments beat the same policy trained only on the target robot's own data. Unrelated tasks still teach shared structure: how objects move, how contact resolves, what a gripper occlusion looks like.

The corollary is a diminishing-returns curve most teams underestimate. Early demonstrations of a task are worth a lot. By the time you have a few hundred clean successes on the same setup, each additional one teaches the model almost nothing it did not already know. That budget would have bought far more capability spent on a new object, a new room, or a new failure mode.

The marginal demonstration is worth exactly what it teaches the model that the earlier demonstrations did not. Past a point, another copy of a solved task teaches close to nothing.

Reading datasets: hours versus coverage

Once you start counting coverage instead of hours, public datasets sort themselves differently. A large single-task log can carry less usable information than a smaller collection deliberately spread across scenes, objects, and viewpoints. The table below lists reported figures for several widely used datasets, chosen to show how different the design choices are.

Note what the last two rows are: human video, not robot logs. Datasets like Ego4D and Ego-Exo4D carry no robot actions at all, yet they cover a breadth of objects, hands, and environments that no teleoperation program can match on cost. Their role in a training mix is exactly the coverage argument in another form, cheap breadth to complement expensive, precise robot demonstrations.

Where the useful bits of information hide

"Diversity" is too blunt a word for what actually matters. A few specific axes carry most of the value:

• Objects and scenes: new instances, clutter, distractors, and backgrounds the policy has never had to ignore.
• Viewpoint and embodiment: different camera placements and robot bodies, which force the model to learn what is invariant rather than memorize one geometry.
• Lighting and time of day: the cheapest diversity to collect, and among the most common causes of a lab-to-field regression.
• Failure and recovery: demonstrations that start from a mistake and return to a good state. These teach the policy the one thing clean successes cannot, how to get back onto the distribution after it drifts off.

Recovery data is the most undervalued of the four. Physical Intelligence and others building generalist policies have argued that robustness comes less from flawless demonstrations than from data that shows the robot how to be wrong and correct itself. A dataset of only perfect runs is, in a sense, a dataset with a hole in exactly the region where deployed robots spend their hardest seconds.

Diversity has a cost function too

None of this means entropy for its own sake. You cannot pour arbitrary data into the mix and expect gains. Mislabeled trajectories, sloppy teleoperation, and demonstrations from an embodiment too far from the target can add noise faster than signal. Diversity and quality are not opposites, they are allies, and the thing both oppose is raw hours of near-duplicate data. The real objective is coverage of the conditions the robot will actually face, sampled cleanly, labeled honestly, and weighted toward the long tail rather than the easy center.

That reframes the collection problem. The question is not "how many hours did we log" but "which slices of the deployment distribution do we still not cover, and what is the cheapest honest way to fill them." Sometimes the answer is more robot teleoperation. Often it is human demonstration, simulation, or simply a more deliberate sampling plan for data you were going to collect anyway.

The teams shipping the most capable robots are not the ones with the largest raw logs. They are the ones who can tell you, for any given skill, which slice of their data taught it and which slice is still missing. Count coverage, not hours.