What contact-rich manipulation demands from data
Force and tactile signals, not pixels, decide contact-rich manipulation. Why RGB can never supply them, and why touch data is the real bottleneck.
Watch a robot hand close on a ripe strawberry. The camera tracks the approach cleanly: the fingers spread, the fruit sits in frame, the gripper descends. Then the fingertips touch, and the pixels that matter disappear. The contact patch is now hidden behind the very fingers making contact. Whether the hand settles into a firm hold or keeps closing until the fruit ruptures is decided in a window of a few milliseconds, from a signal the camera never recorded: force.
For pick-and-place in open space, RGB is close to sufficient. It reports where objects are, how they are oriented, and where the gripper should go. What it cannot report is how hard. The moment a robot presses, wipes, inserts, twists, or pulls against resistance, the decisive variable moves from geometry to mechanics, and mechanics does not emit photons. This is the blind spot at the center of contact-rich manipulation, and it is not one that higher resolution or a bigger vision transformer will close.
Most of the last three years of robot learning has been built on vision and joint positions, because that is what the large datasets contain. Open X-Embodiment stitched together demonstrations from more than 20 labs; DROID pushed manipulation collection across a wide range of scenes and buildings. Both are overwhelmingly visual. That is a reasonable place to start and a poor place to stop. This piece is about what contact-rich work actually demands from a training corpus, and why force and tactile channels are load-bearing rather than optional.
Where the camera goes blind
Three failure modes recur. The first is occlusion: at the instant of contact, the manipulator itself covers the interface, so the most informative region of the image is the one you cannot see. The second is latency. Visible deformation lags the force that caused it, so a policy trained on pixels reacts after the mechanical event rather than during it. The third is ambiguity: a rigid part resting in a fixture and the same part being jammed at 30 Newtons against a misaligned hole can look identical for many frames. Vision reports the consequences of force, slowly and indirectly. It does not report force.
None of this is fixed by a better camera. Depth helps you reach the surface and then goes quiet the moment the surface is touched and occluded. Higher frame rates narrow the latency gap but never state magnitude. You can label a crushed strawberry after the fact; you cannot recover, from any RGB frame, the 20-millisecond ramp of grip force that decided its fate.
What contact-rich actually means
Contact-rich tasks share a signature: success depends on regulating force through a sequence of makes and breaks of contact, often at tolerances tighter than the vision system can resolve. Peg and connector insertion. Wiping and scraping a surface. Cutting. Folding cloth. Threading a cap onto a bottle. Seating a gear. Toyota Research Institute has been explicit that its Large Behavior Models target exactly this dexterous, contact-heavy regime, and Physical Intelligence has shown generalist policies on tasks like folding laundry, where the object changes shape under the hand.
These are not edge cases. They are most of the economically interesting work: assembly, wiring and cabling, food handling, packaging, personal care. The common thread is that geometry gets you to first contact, and everything after that is mechanics.
The signals RGB can never supply
The point is not that vision is weak. It is that each channel has a domain, and force and touch cover the domain where contact-rich value lives. The table below is deliberately blunt about where each modality stops being useful.
| Channel | Reports well | Native rate (order of) | Blind spot |
|---|---|---|---|
| RGB video | Object identity, pose, scene layout | 30-60 fps | Force magnitude, contact state, slip onset |
| Depth | Geometry, distance to contact | ~30 fps | Anything past first touch, once occluded |
| Joint proprioception | Arm configuration, gross load | 100-1000 Hz | Fingertip forces, local slip |
| Wrist force-torque | Net contact wrench, insertion resistance | ~500-1000 Hz | Where on the hand the force acts |
| Fingertip tactile | Local pressure, shear, incipient slip | 100 Hz to low kHz | Global scene, object identity |
A policy that has only ever seen contact cannot feel when it has pressed too hard. By the time the strawberry is visibly crushed, the label is already three frames late.
Why the big datasets under-represent touch
The scarcity is structural. Ego-Exo4D and the earlier Ego4D captured thousands of hours of human activity, a remarkable egocentric vision corpus, but from head-mounted cameras with essentially no calibrated force at the hands. Open X-Embodiment and DROID pair vision with proprioception, which reports arm configuration, not fingertip contact. RH20T is one of the few large collections that deliberately pairs multi-view vision with force-torque and tactile readings, which is precisely why it stands out. The result is a field with an enormous visual corpus and a thin, fragmented record of touch.
This matters because a model can only learn from channels the data actually contains. A vision-heavy corpus produces vision-shaped policies. No amount of scaling turns a dataset that never measured force into one that teaches force.
What good contact data requires
Recording touch well is harder than bolting on a sensor. Four requirements separate a usable contact corpus from a noisy one.
- Rate. Force-torque and tactile events unfold on the order of hundreds of hertz to a kilohertz, while video runs at 30-60 fps. Capture has to be genuinely multi-rate, not force downsampled to match the camera.
- Calibration. A tactile reading in raw sensor counts is close to useless. It has to map to physical units, pressure and shear in Newtons, or one lab's data cannot transfer to another lab's robot.
- Synchronization. Force at 1 kHz and video at 30 fps only teach if their timestamps align to within a frame or better. A slip event and the pixels around it must refer to the same instant.
- Regime coverage. The useful information lives in the transitions: pre-contact, the make of contact, steady hold, incipient slip, release. Data that only samples the steady state omits the moments the policy most needs.
The architectures are ready for touch. Gemini Robotics, NVIDIA GR00T, and Physical Intelligence's π0 are all multimodal policies that can, in principle, condition on force and tactile streams. What they lack is not capacity but experience: demonstrations where the hard-won knowledge of how much to press is written into the data instead of left on the cutting-room floor with the occluded pixels. Record the force, and contact-rich autonomy stops being a demo and starts being a distribution you can train on.