Data Curation for Robot Foundation Models
Data curation, not raw volume, decides how a robot foundation model generalizes: what to keep, what to cut, and how to weight a training corpus.
Two teams start from the same checkpoint. Same architecture, same compute budget, same week of GPU time. One team's policy generalizes to a new kitchen; the other's flails the moment a mug moves two inches. The difference was not the model. It was the corpus each team trained on, and what each team threw away.
This is the uncomfortable truth of robot foundation models. The architecture is increasingly commoditized. The data mix is not. Data curation, deciding what to keep, what to cut, and how to weight the rest, is where most of the real engineering now happens, and it rarely shows up in the demo video.
Curation sounds like janitorial work. It is closer to portfolio construction. Every trajectory you add changes the distribution the model comes to believe in.
More Data Made the Policy Worse
The naive assumption is that a robot corpus behaves like a text corpus: pour in more tokens, watch the loss drop. It does not hold. Robot data is heavily imbalanced by construction. A teleoperation fleet running the same pick-and-place cell for a month produces thousands of near-identical grasps and almost no rare events.
Train on that raw pile and the policy overfits to the common case. It becomes excellent at the grasp it has seen ten thousand times and helpless at the one it has seen twice. Adding a million more easy demonstrations does not fix the failure. It buries the signal that would.
Aggregated corpora make the imbalance concrete. When the Open X-Embodiment effort pooled data across more than twenty robot types, a handful of datasets dominated the token count while long-tail behaviors stayed rare. Training a single policy on the raw mixture underperforms until you reweight it. The lesson is blunt: the mix matters more than the mass.
A robot corpus is not a pile you grow. It is a distribution you shape, and every demonstration you keep is a vote for what the model believes normal looks like.
What Curation Actually Decides
Three decisions sit under the word curation, and they are not equally easy. Keep is the first. Cut is the second. Weight is the third, and it is the one teams underinvest in.
Keeping is about coverage. You want the corpus to span the states a deployed robot will actually see, including the awkward ones: bad lighting, cluttered tables, half-open drawers. Cutting is about hygiene: corrupted timestamps, dropped force channels, demonstrations where the operator gave up halfway. Weighting is about emphasis. Two demonstrations can both be clean and both be worth keeping, yet deserve very different influence on the gradient.
These three pull against each other. Maximize coverage and you readmit noise. Maximize hygiene and you risk throwing out the messy, off-nominal demonstrations that teach a policy how to recover. Balance the mix perfectly on paper and you can still starve a skill that only looks rare because it is expensive to collect. Curation is a set of tradeoffs, not a checklist, and the right point on each axis depends on what the model will be asked to do.
| Signal in the raw data | Decision | What it protects against |
|---|---|---|
| Rare recovery from a failed grasp | Keep and upweight | Brittleness when the policy is off-distribution |
| Ten thousand near-identical easy grasps | Keep a sample, downweight | Overfitting to the common case |
| Dropped force-torque channel mid-episode | Cut the episode | Silent gaps that teach a wrong contact model |
| Duplicate replays of one demonstration | Deduplicate | Inflated dataset size, misleading evaluation |
| Under-represented skill, for example pouring | Upweight or commission more | A capability hole invisible on the aggregate metric |
Weighting: The Lever Nobody Publishes
Reweighting a corpus is the cheapest capability lever available, and the least documented. You do not need new hardware or new demonstrations. You need to change how much each slice of data counts.
The methods run from crude to careful. The crudest is per-dataset scaling: cap any one source so it cannot drown the others, roughly what aggregated benchmarks do before training a cross-embodiment model. Finer approaches weight by skill, by difficulty, or by how much a demonstration reduces the model's error. Groups building generalist policies, including the teams behind NVIDIA Isaac GR00T and Physical Intelligence, treat the data recipe as a first-class part of the model, not an afterthought.
There is a catch. Upweighting rare data amplifies its noise too. One mislabeled recovery, counted five times, teaches the wrong lesson five times as loudly. Weighting and cleaning are not separate chores. Do them together, or the leverage cuts the wrong way.
Curation Is Only as Good as Your Metadata
Every decision above assumes you can tell your slices apart. You cannot upweight recoveries you never labeled. You cannot cut episodes with a dropped channel if nobody recorded which channel dropped. Curation lives downstream of capture, and it inherits whatever the capture pipeline bothered to write down.
This is why serious corpora such as the DROID dataset ship rich per-episode metadata, scene, embodiment, camera setup, and why the teams building large behavior models at Toyota Research Institute treat provenance as part of the dataset rather than a nicety. Without that structure, curation degrades into guesswork over one giant undifferentiated blob.
Evaluation Inherits Your Curation
Curation does not only shape training. It shapes the scoreboard. If your held-out test set is drawn from the same over-represented easy grasps, a policy can post a high number and still fail in the field. Near-duplicate demonstrations that leak across the train and test split make it worse, because the model has effectively seen the exam. This is why deduplication and a deliberately hard, coverage-driven evaluation set belong to curation, not a separate afterthought. The number you trust is only as honest as the split it came from.
The Recipe Is the Product
It is tempting to treat curation as cleanup you do once the interesting modeling work is finished. That gets the order backwards. For robot foundation models, the corpus and its weighting are the product decisions with the longest half-life. Architectures will churn. A well-curated, well-documented corpus keeps paying off across model generations.
So the question to ask a data vendor is not how many hours they hold. It is how they know what is in those hours, and what they would cut. If they cannot answer, they are selling mass, not signal.