Tactile Sensing: The Missing Modality in Robot Data
RGB video goes blind the instant a gripper touches an object. Here is what tactile sensing and force data add to manipulation, and why they stay scarce.
A robot arm lowers its gripper onto a paper cup half full of coffee. On camera the grasp looks clean: the fingers close, the cup lifts, the wrist rotates toward the tray. Then the cup slides a few millimeters, the controller squeezes harder to compensate, and the wall of the cup buckles inward. The whole failure was recorded at high resolution and high frame rate. Almost none of it was seen, because the variables that failed, the shear force at the fingertip and the first micrometers of slip, never landed on a pixel.
Nearly all of the data now training manipulation policies is built from cameras. Open X-Embodiment stitched together dozens of robot datasets, and the shared signal across them is RGB plus proprioception. DROID scaled teleoperation with multi-view RGB-D. First-person human corpora such as Ego-Exo4D capture, in fine detail, what hands do from the wearer's point of view. These are the substrate for the vision-language-action (VLA) models everyone is now training. They also share one blind spot, and it is the one that matters most for manipulation: contact.
Touch is not a bonus channel to bolt on later. It is the modality that carries the physics of a grasp. RGB is a measurement of light bouncing off surfaces; the moment two surfaces meet, the informative variables stop being photons and start being forces. Put bluntly, in contact-rich work the useful signal moves off-camera at exactly the moment the task gets hard.
What a camera stops seeing at the moment of contact
Vision is excellent up to the millisecond before contact. It localizes the object, plans the approach, and classifies what is being grasped. Then three things happen at once, and a camera catches none of them well.
- Occlusion. The fingers close over the exact patch you care about. The contact interface is now hidden behind the hand that formed it.
- Force is not a visual quantity. A gentle hold and a crushing one can look identical from the outside. You cannot reliably read 5 N from 15 N off pixels, and stiffness, the property that tells a cup from a can, stays invisible until something deforms.
- Slip is fast and small. Incipient slip, the partial micro-sliding that precedes a full slip, is a sub-millimeter, sub-tenth-of-a-second event. By the time it is visible in video, the object is already moving.
The physics that force data encodes
Where vision goes quiet, contact sensing gets loud. A tactile or force channel reports the quantities that actually govern whether a grasp holds:
- Normal and shear force, plus torque about the contact, which together decide whether the object stays put or rotates out of the grasp.
- Incipient slip, which lets a controller raise grip force just enough, the way a person adjusts within roughly a tenth of a second of a cup starting to go.
- Compliance and texture below optical resolution: how hard a surface is, how much it gives, whether it is slick or grippy.
The hardware to capture this is real and diverse. Vision-based tactile sensors such as GelSight and DIGIT turn a soft gel and a tiny camera into a high-resolution map of the contact surface. Force-torque sensors at the wrist read the six-axis wrench on the whole hand. Toyota Research Institute's soft "bubble" grippers made deformable tactile sensing a load-bearing part of a manipulation stack rather than a lab curiosity. Wrist force-torque loops run near a kilohertz, an order of magnitude faster than a typical camera stream, while camera-based tactile sensors sample at video rates because their sensing element is a camera.
Vision tells a robot where to reach. Only contact tells it whether the reach worked.
Two modalities, side by side
Laid out plainly, the two channels are not competitors. They are complements that peak at different instants of the same motion, vision before contact and force during it.
| Quantity | RGB video | Contact and force sensing |
|---|---|---|
| Object location and shape | Strong | Partial, only where touched |
| Grip force magnitude | None | Direct |
| Incipient slip | Effectively none | Direct, sub-100 ms |
| Material stiffness | Inferred, unreliable | Measured on contact |
| State during finger occlusion | Lost | Preserved |
| Typical sampling rate | Tens of Hz | Hundreds to ~1000 Hz |
Why the contact data barely exists
If force is so central, why is almost none of it in the big training sets? The answer is mundane and structural: contact data is hard to collect and harder to standardize.
Cameras are cheap, interchangeable, and speak a common format. Tactile sensors are none of those. There are many incompatible form factors, they drift and wear, calibration is per-unit, and there is no shared schema for what a "touch reading" even is across two labs. Teleoperation rigs, the workhorse of modern collection, rarely stream calibrated force alongside video. And the largest human demonstration source of all, first-person video, has no force ground truth whatsoever: Ego-Exo4D knows where a hand is, not what it feels.
A few datasets push against this. RH20T deliberately pairs multi-view vision with force-torque and audio, so a policy can learn what a contact event sounds and feels like, not just how it looks. On the model side, Toyota Research Institute's large behavior models and the dexterity work at Physical Intelligence both treat rich, closed-loop contact behavior as the target rather than an afterthought. These remain the exception. The default corpus is still overwhelmingly visual, which means many policies quietly learn to guess contact from visual priors, and that guess falls apart on a new object, a new material, or a bad grasp.
What changes when force is in the training set
Contact is also where simulation hurts most. Stiff contacts, friction, and deformation are the least reliable part of any physics engine, so a policy trained purely in sim inherits a contact model that does not match the world. Real, calibrated force data is the anchor that pulls those policies back toward reality, and it is a signal a VLA backbone such as NVIDIA's Isaac GR00T has the capacity to absorb if the data exists to feed it.
The payoff shows up precisely on the tasks that embarrass vision-only systems: inserting a plug, seating a connector, handling a deformable bag, using a tool whose feedback is felt rather than seen. These are not edge cases. They are most of the work a humanoid would be asked to do in a home or a warehouse.
The camera problem in robotics was mostly solved by scale. The touch problem will not be, because the data to solve it is not sitting on the internet waiting to be scraped. It has to be captured on purpose, with the forces measured and aligned to everything else. The teams that treat contact as a first-class modality now, rather than a channel to add once the vision pipeline works, are the ones whose robots will still be holding the cup a year from now.