What Counts as a Good Demonstration?

Two people record the same demonstration: pick up a mug, set it on a coaster. One recording makes the policy better. The other quietly makes it worse. Same task, same robot, same frame count. The difference is quality, and quality in a demonstration is not one property but several, each able to fail on its own.

Robot learning has spent years chasing scale: more hours, more trajectories, more teleoperation. Scale matters. Yet a large pile of mediocre demonstrations can train a worse policy than a small set of clean ones. Behavior cloning imitates what it is shown, faithfully, mistakes included. So the question sitting under the whole field is deceptively plain: what counts as a good demonstration?

A policy imitates, so it inherits your habits

Behavior cloning maps observation to action. It has no idea which of your movements were skillful and which were sloppy. If your teleoperated grasp hesitated, the model learns to hesitate. If you overshot and corrected, it learns the overshoot and the correction as a single motion, and it will reproduce both. This is the core of imitation learning: the demonstration is the teacher and the ceiling at once. Work from the Physical Intelligence blog and the open Hugging Face LeRobot stack keeps circling the same lesson, that policy quality tracks demonstration quality far more tightly than raw dataset size.

There is a subtler trap here too. A demonstration can succeed and still teach the wrong thing. Grab the mug by luck after three failed swipes, and the policy learns the swipes as part of the recipe. Success is necessary. It is nowhere near sufficient.

The quality dimensions of a single demo

It helps to break "quality" into dimensions you can inspect one at a time. Any single row below, left broken, is enough to turn a demonstration from an asset into a liability.

These dimensions are not independent knobs. Coverage without consistency gives you noise; consistency without coverage gives you a policy that memorized one pose. The craft is holding several of them at once, demo after demo, operator after operator.

Modal richness deserves its own warning. A demonstration missing its force channel is not simply a smaller demonstration; for a contact-rich task it is a misleading one, because the policy learns to reach the right pose while staying blind to how hard it pressed. Depth without force, force without alignment, video without timestamps: each gap turns a demo from partial into actively wrong for some slice of tasks. A modality you captured but never synchronized is often worse than one you never captured, because it looks trustworthy.

Why quality compounds instead of averaging

Bad demonstrations do not average out. A policy trained on clean data that errs only occasionally still drifts away from the states it saw, and injected bad demos widen the action distribution in unpredictable ways. Rare skills are the most exposed. A skill might have only a few dozen demonstrations, so a handful that are laggy or mislabeled can dominate what the model learns for it. Near-duplicates make it worse by inflating the count: a corpus can look ten times larger than it is while teaching almost nothing new. This is why teleoperation corpora like DROID and aggregations like Open X-Embodiment invest so heavily in a uniform capture protocol. The protocol is not bureaucracy. It is the thing that keeps one operator's bad habits from quietly rewriting a shared dataset.

Filtering after the fact helps less than people hope. You can throw out the obvious failures, but the quiet damage comes from demonstrations that succeeded through a bad strategy, and those look fine to an automated filter. A jerky, hesitant grasp that still landed the mug passes every success check and still teaches the policy to hesitate. Catching it takes either a human eye or a consistency model that already knows what the good version of the skill looks like, which is itself built from good demonstrations. The problem is recursive, and it rewards getting capture right the first time.

Judging a demo at capture time

The cheapest place to catch a bad demonstration is the moment it is recorded, not months later in a cleaning pass. A short field checklist does most of the work. Did the run succeed on the first attempt? Is the operator's action stream tightly aligned to the observation stream, with no latency between what they saw and what they commanded? Is the strategy consistent with the rest of the batch? Does the demo add coverage, or is it a near-copy of the last one? Are all modalities present and synchronized? Any "no" means the demo gets fixed or dropped before it ever reaches training. Cleaning later is possible, but it is slow, lossy, and easy to skip under deadline.

Consistency across operators is the dimension teams underestimate most. Ten people asked to pour a glass of water will hold the bottle at ten angles, approach from different sides, and stop at different fill levels. Some of that variation is healthy coverage. Too much, and the policy sees a task with no stable structure and learns an averaged blur that satisfies no one. The fix is not a rulebook that kills variation. It is a shared reference for what the skill's core actually is, so variation lands around a center instead of scattering.

Scale decides how far your policy can go; quality decides whether it is worth starting. Behavior cloning cannot rise above the demonstrations it is fed.

Quality first, then scale

The road to a reliable robot is not paved by shoveling more mediocre demonstrations into the training set. It is walked by teams that can tell good from bad and reject the bad at the point of capture. Scale still counts, but it is a multiplier, not a substitute. Get the single demonstration right, then multiply by volume, and you compound. Reverse the order and all you scale is noise.