Multimodal Robot Data: Force, Audio, Depth Beyond RGB

Plug a USB-A connector into a port you cannot see. You miss, feel the lip of the socket, rotate the connector a few degrees, feel the tongue drop in, and push. Under two seconds, and almost none of the useful information in that recovery reached your eyes. It arrived through your fingertips as force, through the small click as sound, and through a geometry your hand already carried.

Most robot-learning datasets would have logged that moment as a video clip and a stream of joint angles. That is enough to train a policy to reach toward the port. It is not enough to teach the policy what to do when the connector binds, because the binding never appears in the pixels. Contact-rich manipulation, insertion, wiping, peeling, cutting, buttoning, folding, is dominated by signals a camera does not carry.

This is the case for capturing more than RGB. What force, audio, and depth each add, where each one fails, and why the genuinely hard engineering is not the sensors but keeping their clocks in step.

What a camera cannot see

An RGB frame is a projection. It reports where surfaces land in the image, at whatever rate the shutter runs, usually 30 to 60 Hz. It says nothing about how hard two surfaces press together, whether a grasp is beginning to slip, or how far away anything sits in meters. For pick-and-place in free space that omission rarely bites; a policy can recover from the visual stream alone. The Open X-Embodiment collection, which pools more than a million trajectories from 20-plus labs, is overwhelmingly RGB plus proprioception, and it still trains capable reaching and grasping behavior.

Contact changes the problem. The instant two rigid bodies touch, the informative signal migrates from vision to mechanics. A peg entering a hole with 0.1 mm of clearance produces jamming forces that a 30 Hz camera cannot resolve in time and, inside a tight fixture, often cannot see at all. The policy that solves that insertion is reacting to force, not to pixels. Train it on video alone and you have taught it the approach while hiding the part that actually fails.

Force and tactile: the signal the task is about

Two different signals hide under the word touch. Wrist force-torque is a 6-axis reading, three forces and three torques measured at the robot's wrist, and it reports the net load on whatever the hand holds. Tactile sensing is distributed across the fingertips and reports the pressure field at the contact patch itself, often through a vision-based sensor such as a GelSight-style elastomer that photographs its own deformation. The first tells you that you are pushing; the second tells you where, and whether you are about to slip.

Both have to be fast. Contact events are brief and stiff: a latch seating, a screw catching a thread, a grasped object starting to slide, each can play out in a few milliseconds. Sample force at 30 Hz and you alias the very transient you care about, smearing a sharp event into a blur. Practical rigs read wrist force-torque in the hundreds of hertz up to a kilohertz, an order of magnitude above the video rate. RH20T, a manipulation dataset built specifically around contact-rich skills, records wrist force-torque and audio alongside RGB-D for exactly this reason.

A camera can watch a peg approach a hole. It cannot feel the peg bind, and the policy that solves insertion is reacting to what the camera missed.

Audio: the modality most teams skip

Audio is the cheapest modality to add and the one most often left off the rig. Contact makes sound, and the sound is frequently a better contact detector than force. The click of a buckle, the scrape of a sponge across dried food, the seating of a screw, the hollow knock that says a container is empty: each is a sharp acoustic event with an unambiguous onset. A contact microphone sampling at 44.1 or 48 kHz captures a transient that a 30 Hz camera drops entirely.

Academic robotics labs have repeatedly shown audio carrying real task signal, from telling materials apart to pinning the moment of contact more reliably than vision does. The practical catch is that the useful part of robot audio is mostly structure-borne, so a microphone coupled to the tool or the wrist beats a room mic that is swamped by fans, HVAC, and bystanders.

Depth: metric geometry, and where it breaks

Depth removes the scale ambiguity baked into a single RGB frame. An RGB-D sensor hands you metric geometry directly, which helps grasp planning, collision checking, and any policy that has to place an object at a real distance rather than a pixel offset. DROID, an in-the-wild teleoperation dataset of roughly 76,000 trajectories, pairs stereo and depth with its RGB so the scenes are grounded in metric space rather than left as flat images.

Depth is not free of failure, and it fails on the things manipulation cares about most. Structured-light and time-of-flight sensors return noise or empty holes on transparent glassware, specular metal, thin edges, and dark matte surfaces. Stereo needs texture to match and goes blind on a blank wall. And depth describes geometry, not contact; it falls silent at the exact instant two surfaces meet, which is the instant the task turns hard.

What each modality buys you

No single channel is sufficient, and none is free. The tradeoff is concrete.

The hard part is time, not sensors

Bolting six sensors to a rig is a weekend of work. Making their data trustworthy together is the actual project. Each stream arrives on its own clock: video near 30 Hz, depth near 30 Hz, force at a kilohertz, audio at 48 kHz, joint encoders somewhere between. If a contact event lasts 5 ms and your force and video timestamps disagree by 20 ms, no model can learn that the force spike and the visual event are the same instant. It learns a false offset instead, and a false offset is worse than a missing channel.

So the real deliverable of a capture stack is not the list of modalities; it is sub-frame temporal alignment across all of them, plus honest per-channel rate labels so a training pipeline knows what it is resampling and by how much. Ego-Exo4D, which pairs egocentric and third-person video with spatial audio across over a thousand hours, is as much an exercise in multi-stream synchronization as in collection. Get the timestamps wrong and every extra modality is worse than useless, because each one now teaches the model correlations that were never there.

None of this is an argument against RGB. Vision stays the backbone; it is how a policy finds the port in the first place. The claim is narrower and harder to dodge. The tasks that still defeat robots are the contact-rich ones, and those tasks live in the signals a camera was never going to record. Capture them, aligned to the millisecond, or keep training policies to reach for a connector they will never quite feel drop in.