Temporal Alignment of Multimodal Streams

A robot reaches for a bottle, closes its gripper, and lifts. On the video it looks perfect. In the training data, the force spike from contact was logged 40 milliseconds before the frame that shows the fingers touching the glass. To the model, force now precedes touch. It learns to squeeze early, and in deployment it crushes the bottle before it has hold of it.

Nobody wrote a bug. Every sensor worked. The cameras ran at 30 fps, the force-torque sensor at 1 kHz, the joint encoders somewhere in between, and each stream carried its own clock and its own latency. The failure lived in the gaps between them.

Temporal alignment, getting every modality onto one shared timeline, is the least glamorous part of building a multimodal robot dataset. It is also the part that quietly decides whether the data teaches physics or fiction.

Why Milliseconds Decide the Policy

Manipulation is a game of contact events, and contact is where timing turns unforgiving. A grasp succeeds or fails in a window of tens of milliseconds. The instant a fingertip touches a surface, the force signal jumps, the visual scene barely changes, and the correct action flips from close to hold.

If the force stream and the video stream disagree about when that instant happened, the model learns a warped cause and effect. Vision-heavy tasks tolerate a little slop. Contact-rich tasks do not. This is why datasets aimed at contact, such as RH20T, treat synchronization across vision, force, and proprioception as a first-order design constraint rather than a detail.

Audio deserves a mention here, because it is the modality teams forget to align. The click of a latch, the scrape of a box across a table, the snap of a connector seating: these are precise, informative contact cues, and they sit on a completely separate clock inside the sound card. Aligned well, audio can pin the exact moment of contact more sharply than vision does. Aligned badly, it is one more stream teaching a false sequence of events.

In contact-rich manipulation, a 40 millisecond misalignment does not add noise. It teaches the wrong causal order, and the policy learns it faithfully.

Where the Clocks Drift

Misalignment is not one problem. It is a family of them, and they stack.

• Different native rates. A camera at 30 fps samples every 33 ms. A force sensor at 1 kHz samples every 1 ms. Line them up and most force samples have no matching frame.
• Sensor latency. A USB camera exposes, encodes, and transfers before its frame arrives. That pipeline delay can be tens of milliseconds, and it differs from device to device.
• Clock domains. Each device may timestamp against its own oscillator. Two clocks that both read seconds still drift apart over a long session.
• Buffering and drops. A dropped frame or a full buffer shifts everything after it, if you index by count instead of by time.

Any one of these is manageable. Together they produce an offset that is neither constant nor obvious, and that changes from session to session.

Resampling is the usual response, and it hides its own trap. To train on aligned tensors, teams commonly resample every stream to one nominal rate. Downsample a 1 kHz force signal to match 30 fps video and you throw away the very spikes that mark contact. Upsample the video and you invent frames that were never seen. Neither move is wrong in itself, but both are lossy, and the loss is largest exactly where manipulation is decided. The safer default is to keep each stream at its native rate and carry explicit timestamps, so a downstream user can resample deliberately rather than inherit a choice they cannot undo.

Hardware Sync Beats a Software Guess

There are two ways to put streams on one timeline, and they are not equal. Software timestamping records the arrival time of each sample on a shared host clock. It is cheap, and it is always at least a little wrong, because arrival time is not capture time.

Hardware synchronization ties devices to a common trigger or clock signal, so shutters fire together and samples share a reference. It is more work to build, and it is the only approach that survives contact-rich data. Multi-view capture efforts like Ego-Exo4D lean on careful cross-camera synchronization for exactly this reason, and the manipulation datasets pooled in Open X-Embodiment vary widely in how tightly their streams are aligned, which is one reason mixing them is harder than it looks. Large teleoperation corpora such as the DROID dataset document their camera setup precisely so others can reason about alignment at all.

The offset you can measure, you can fix

A known, constant latency is not a problem. You subtract it. The dangerous offsets are the variable ones: jitter in the control loop, a clock that drifts a millisecond per minute, a buffer that occasionally hiccups. Those cannot be fixed with one constant. They have to be measured continuously and logged per frame, which only works if the capture rig was built to record its own timing.

Trust the Timeline, or Trust Nothing

Multimodal data is sold on the promise that force, vision, and proprioception describe the same moment. If the timeline is wrong, that promise is empty, and the extra modalities become worse than useless: they teach confident errors. Alignment is not a postprocessing checkbox. It is the thing that makes a multimodal dataset multimodal, instead of several unrelated recordings that happen to share a folder.

So when a dataset advertises synchronized streams, ask how. Hardware trigger or software timestamp. Per-frame timing or a single nominal rate. The answer tells you whether you are buying physics or fiction.