How teleoperation data is collected, end to end

A good teleoperation session for a laundry-folding policy might run four hours and produce ninety usable minutes. The rest is the operator repositioning between attempts, a dropped stereo frame, a gripper that reported closed while the towel slipped through it, or a demonstration that technically succeeded and quietly taught the model a habit nobody wanted. Collecting robot demonstration data is less like filming a video and more like running a small, unforgiving recording studio in which five clocks disagree and the talent is a 7-DOF arm.

This is the layer of the robot-learning stack that rarely makes the announcements. The models get the headlines: NVIDIA Isaac GR00T, Physical Intelligence's pi-0, Toyota Research Institute's Large Behavior Models, Google DeepMind's Gemini Robotics. Every one of them is a function of the trajectories it was trained on, and the overwhelming majority of those trajectories begin with a person deliberately moving a robot through a task. What happens between the rig and the labeled trajectory is where most of the quality is decided.

Walk it end to end and the same lesson keeps surfacing: the hard problems are not in the machine learning, they are in timekeeping, alignment, and ruthless filtering. Here is the pipeline.

The rig decides what you can even record

Teleoperation means a human drives the robot in real time while every channel is logged. How the human drives it sets a ceiling on the data before a single frame is saved. Four families dominate.

Leader-follower puppeteering pairs a small replica arm with the real one; the operator moves the leader and the follower mirrors it. ALOHA popularized this for bimanual manipulation, and GELLO showed you can build a usable leader arm from a few hundred dollars of 3D-printed parts and servos. Motion feels direct and the joint mapping is clean. VR and handheld controllers free the operator from a physical replica; the widely used DROID dataset was collected with an Oculus controller driving a Franka arm across dozens of buildings. Gloves and exoskeletons capture finger-level dexterity and can feed force back to the operator, at the cost of calibration and per-user fitting. Kinesthetic teaching, physically guiding the arm by hand, needs no interface hardware but records no natural camera viewpoint and does not scale.

Every stream has its own clock

A single demonstration is not one recording, it is a bundle of streams sampled at different rates. RGB cameras run at 30-60 Hz. Joint encoders and end-effector proprioception report at 100-1000 Hz. A wrist-mounted force-torque sensor can run faster still. The teleoperation commands arrive whenever the operator moves. None of these are naturally in step, and the mistake that ruins a dataset is assuming they are.

The fix is boring and non-negotiable: timestamp every sample at its source against one shared clock, then resample onto a common timeline afterward rather than trusting arrival order. Multi-camera rigs like the stereo-plus-wrist setup in DROID have to be time-synchronized and geometrically calibrated, or the depth and the action will disagree. The subtler trap is action-observation alignment. The operator reacts to what they saw a beat ago, so the raw log pairs each action with an observation the human had not yet seen when they moved. Train on that off-by-one pairing and you teach the policy to act on the future. Correcting for end-to-end latency, often tens of milliseconds, is part of building the trajectory, not an optional refinement.

A demonstration dataset is a synchronization artifact first and a behavior dataset second. Get the clocks wrong and every downstream label inherits the error.

Most demonstrations are quietly broken

Assume a meaningful fraction of raw sessions are unfit to train on, and build the pipeline to find them. The failures are rarely dramatic. A camera drops frames under USB bandwidth pressure. A clock drifts past its tolerance mid-episode. The gripper state reads closed while the object was never actually grasped. The operator pauses to think and leaves thirty seconds of near-static frames. A force spike marks a collision the operator recovered from but never flagged.

Serious collection runs a two-stage filter. Automatic checks catch the mechanical faults: frame-drop counts, timestamp gaps, proprioception that disagrees with the commanded motion, force readings outside a sane envelope, and episodes too short or too idle to carry signal. Human review handles judgment: was the task actually completed, was the strategy the one you want taught, is the language instruction (fold the towel in half) an honest description of what the arm did. RH20T, which pairs more than 110,000 robot manipulation sequences with force-torque and multi-view audio-visual data, is a reminder that once you log richer channels you also acquire more ways for a clip to be subtly wrong. Both successful and failed episodes are worth keeping, but only if they are labeled as such.

From raw log to labeled trajectory

The surviving sessions get reshaped into a fixed schema so a training loop can consume them without special cases. An episode becomes an ordered sequence of steps, and each step is a dictionary: one or more camera images, the proprioceptive state, the action taken, a done flag, and usually a natural-language instruction. Two formats dominate. RLDS, the format behind Open X-Embodiment, packs episodes as nested TensorFlow datasets; the LeRobotDataset format on Hugging Face has become the common currency for the open community and ships with the tooling to load, visualize, and share an episode.

Three choices at this stage shape everything downstream. First, the action space: absolute joint angles, or end-effector deltas. Deltas transfer across similar arms more gracefully, which matters the moment you mix embodiments. Second, normalization: actions and states are rescaled per dimension so no channel dominates the loss, and the normalization statistics have to travel with the data. Third, cross-embodiment metadata: Open X-Embodiment stitched over one million trajectories from 22 distinct robot types into one training corpus, and that only works because every episode declares its embodiment, camera layout, and control frequency. A trajectory without that metadata is nearly worthless in a mixed dataset.

The tradeoffs nobody escapes

Teleoperation buys clean, causal, correctly-embodied data, and it stays expensive because a human is in the loop for every second of it. That single fact drives most of the field's structural bets. Physical Intelligence leans on broad, heterogeneous data to make one model span many robots. Toyota Research Institute pools large volumes of teleoperated demonstrations into Large Behavior Models rather than training a policy per task. Others push egocentric human video, Ego-Exo4D among the reference datasets, to sidestep the robot bottleneck, accepting the embodiment gap in exchange for scale. There is no free lane. More scale usually costs fidelity or provenance, and higher fidelity usually costs throughput.

Where the real work lives

The next time a robot policy demonstrates a clean bimanual fold, remember that the impressive part happened months earlier, in a room where someone decided which four of the sixteen hours were honest enough to keep. The rig is the easy part. The clocks, the filters, and the labels are the craft, and they are what turns raw sessions into a corpus a model can actually learn from, instead of a pile of clips that only look like data.