Deduplication in Robot Datasets: Size vs Signal

Two labs each announce a fifty-thousand-demonstration dataset. One is fifty thousand distinct attempts. The other is five thousand real trajectories, each replayed and lightly jittered ten times. On the headline number they look identical. The policies you train on them will not be, and the benchmark scores you report from them may be quietly wrong.

Deduplication is the least glamorous job in dataset work and one of the most consequential. Text and image teams learned this the hard way: near-duplicate documents inflate corpora, waste compute, and, worst of all, leak across the train-test boundary and inflate reported accuracy. Robot datasets have the same disease, in a form that is harder to diagnose.

The claim here: raw demonstration count is a vanity metric, and until a robot dataset reports its effective, deduplicated size, its headline number tells you almost nothing about what a policy will learn from it.

Where the duplicates come from

Robot data manufactures near-duplicates in ways text never does. Teleoperators repeat the same task from the same starting pose dozens of times, because that is how you collect volume. Scripted resets place the object in the identical spot each episode. Data augmentation clones a trajectory with small perturbations, which is useful for training but ruinous if the clones straddle your evaluation split. And when many datasets are pooled, as in Open X-Embodiment, the same underlying trajectories can enter through more than one contributor.

None of these are misconduct. They are the natural exhaust of practical collection. The problem is that they all inflate the count without adding the thing that count is supposed to proxy: coverage of new situations.

There is also an incentive problem. A bigger number is easier to announce and easier to sell, so the pressure runs toward reporting raw totals rather than unique ones. Nobody has to lie for this to happen. The metric that is easy to measure quietly becomes the metric that gets quoted.

Why duplicates hurt twice

The first cost is training waste. A model spends capacity and compute re-learning trajectories it has already seen, and the loss curve looks healthy while the policy learns nothing new. You pay for a large dataset and get the diversity of a small one.

The second cost is worse, because it is invisible. If a near-duplicate of a training trajectory lands in your test set, your evaluation is measuring memorization, not generalization. The reported success rate goes up and the real-world success rate does not move. Teams that write honestly about robot evaluation, including maintainers of open stacks like LeRobot, keep returning to this contamination risk, because a clean train-test split for continuous trajectories is genuinely hard to guarantee.

This also muddies the scaling story. When a dataset doubles and the policy does not improve, the easy conclusion is that scaling has stalled. Often the real cause is duller: the new half was mostly duplicates, so the effective dataset barely grew. Deduplication is what lets you tell a true plateau from a bookkeeping artifact.

The honest unit of a robot dataset is not hours recorded or demonstrations logged. It is the number of genuinely distinct situations a policy can learn from.

Measuring uniqueness is harder than it sounds

In text, a duplicate is often a near-exact string, and hashing catches it. A robot trajectory lives in a continuous, multimodal space: joint states, images, forces, all unfolding over time. Two demonstrations of the same pick can differ in every raw value and still be, for learning purposes, the same example. The reverse also holds. Two visually similar clips can encode different contact dynamics.

So deduplication for robots is really a question of similarity in the right space. Practical approaches embed each trajectory, cluster in that embedding space, and estimate coverage rather than count rows. Some groups reason about state visitation: how much of the relevant state space the dataset actually touches, a recurring theme in recent robotics research. Large teleoperation efforts such as DROID and contact-rich collections like RH20T are useful precisely because they were built to spread across scenes and objects, not to pile up repetitions of one.

It helps to separate two jobs. Exact deduplication removes byte-identical or replayed trajectories, and it is cheap and safe. Semantic deduplication is the hard one: deciding that two different-looking demonstrations teach the same thing. There is no clean threshold for it, and setting the bar too aggressively erases genuine variation. Most teams end up tuning it against a downstream metric rather than in the abstract.

Dedup is not just pruning

It would be a mistake to hear all this as a call to delete your data. Removing every near-duplicate blindly can throw away useful signal, since a handful of repetitions of a hard, contact-rich motion genuinely helps a policy. The goal is not a smaller dataset. It is an honest one: knowing which samples are redundant, weighting them accordingly, and never letting a training trajectory and its twin sit on opposite sides of an evaluation split.

Weighting is usually the better move than deleting. Down-weight a cluster of near-duplicates so it counts roughly once, keep a few copies where repetition genuinely aids a hard skill, and reserve outright deletion for exact replays. The result is a training mix that reflects real coverage rather than collection habits.

Read the number that matters

Next time a robot dataset is announced with a big number, ask a smaller one: how many distinct situations, after deduplication, does it contain? A dataset that can answer is doing the hard part. One that cannot is quoting a vanity metric, and vanity metrics do not survive contact with a real robot. The teams that win will compete on effective diversity, not on the size of the log.