The hidden cost of robot data annotation

Two labs record what they call the same task: lift a mug off a table. On the first rig, the gripper's force sensor and the wrist camera are stamped by one clock. On the second, the camera timestamps come from the USB driver while the force readings come from a separate control loop, and the two clocks drift about 80 milliseconds apart. Feed both into the same model and something subtle breaks. The policy learns that the gripper should start closing slightly before the fingers actually touch the mug, because in the training data the contact signal arrived early. On a good day it looks fine. On a bad day it crushes the cup.

This is the annotation problem, and it is not really about drawing boxes. The expensive, failure-prone part of a modern robot dataset is rarely the motion itself. It is the metadata wrapped around the motion: the language label that says what the trajectory was for, the calibration that says where each camera actually sits, and the time-sync that says which observation caused which action. Get those wrong and the trajectory is still there, still playable, still counted in your headline number, and quietly poisonous to train on.

Headline numbers are the tell. A dataset advertised as "one million trajectories" or "three thousand hours" reports how much was collected, not how much is trustworthy. The gap between those two quantities is where most of the real cost lives, and almost none of it reaches the press release. Three layers deserve most of the blame.

The label nobody agrees on

Language conditioning is now standard. Physical Intelligence's pi-zero, NVIDIA's GR00T line, Google DeepMind's Gemini Robotics, and Toyota Research Institute's Large Behavior Models all take a natural-language instruction and produce actions. That only works if the training data carries instructions, and instructions are where human judgment leaks in.

Consider granularity. "Make a coffee" is one label; "pick up the mug, place it under the spout, press the button" is three; a keystep annotator might produce twenty. Ego4D attached dense free-form narrations to thousands of hours of first-person video, and its successor Ego-Exo4D layered on expert commentary and step-level keysteps precisely because a single caption cannot serve both a high-level planner and a low-level controller. Every level you add is a separate annotation pass with its own labor bill and its own inter-annotator disagreement.

Then there is provenance of the label itself. Some datasets collect the instruction first and then act, so intent is clean but behavior may wander. Others act first and relabel in hindsight, which is cheap and scalable but invites captions that describe what happened rather than what was intended. Both are defensible. Mixing them silently, inside one training set, is how a model learns that "put the block in the bowl" sometimes means the blue block and sometimes the nearest one.

A robot dataset is only as good as its worst-aligned timestamp and its least consistent label. Neither shows up in the trajectory count.

Calibration is a data problem, not a setup step

Calibration is usually treated as something you do once, before the interesting work starts. In a learning dataset it is not a setup step; it is a label, and a fragile one. Camera intrinsics tell you how the lens maps the world to pixels. Extrinsics, including the hand-eye transform between a wrist camera and the gripper, tell you where those pixels sit in the robot's frame. Any 3D supervision, depth alignment, point-cloud fusion, multi-view geometry, is only as accurate as those numbers.

The failure mode is quiet. A wrist camera that is off by two degrees does not throw an error; it produces a dataset where the projected gripper position sits a consistent centimeter from where the gripper really is. Train on enough of that and the policy internalizes the offset. DROID spent real effort standardizing on one arm, the Franka Panda, across 13 institutions, specifically so calibration and control conventions would not vary from lab to lab. That discipline is a large part of why the data composes at all.

Time-sync: the fifty-millisecond lie

Modern rigs are multi-rate by nature. RGB runs near 30 Hz, depth on its own schedule, proprioception faster, force-torque often at hundreds of hertz or more, audio faster still. Each stream carries a timestamp, and the timestamps come from different places: a camera driver, a control loop, a microcontroller, the wall clock. If you align streams by the moment your logger received a packet rather than the moment the sensor sampled the world, you have baked in a latency that varies with system load.

Fifty to eighty milliseconds sounds harmless. For a contact-rich manipulation policy it is not, because it corrupts causality. The model is trying to learn which observation triggered which action, and a misaligned force channel teaches it that contact precedes motion, or follows it, in ways that never happened. RH20T, which pairs vision with force-torque and other modalities across more than one hundred thousand sequences, is fundamentally an exercise in getting those clocks to agree.

What the headline number hides

Four widely used datasets, four different bets on where to spend annotation effort. The numbers below are approximate and rounded.

Why pooling makes it worse

The field's answer to data scarcity is aggregation, and aggregation multiplies every problem above. Open X-Embodiment pooled about 60 datasets across 22 embodiments from 21 institutions into a single training corpus, and the moment you do that, one dataset's "gripper open equals 1" meets another's "gripper open equals 0," one lab's meters meet another's millimeters, and 10 Hz control meets 20 Hz control. None of that is visible in a trajectory count either.

This is why a shared data format, rather than a shared model, is doing quiet heavy lifting. Hugging Face LeRobot pushes a common on-disk schema for episodes, observations, and actions, so the alignment work happens once, at ingestion, instead of being rediscovered by every team that touches the data. It is unglamorous, and it is where a large share of usable robot data now flows.

The next real jump in manipulation probably will not arrive as a new attention variant. It is more likely to come from a team that decided a timestamp is a first-class engineering artifact, that calibration is a label with an error bar, and that a sentence describing a task deserves the same version control as the code that trained on it. Boring work. It is also where the data stops lying.