Proprioception: The Sense Robots Cannot See

Close your eyes and touch your nose. You did not need to see your hand to find it. That sense, the one that tells you where your limbs are without looking, is proprioception, and it runs constantly under everything you do with your body.

Robots have the same sense, and it is arguably their most reliable one. A camera can be blinded by glare, blocked by the robot's own arm, or fooled by a shiny surface. Joint encoders almost never lie. Yet in the rush to train humanoid policies on internet-scale video, proprioception is the channel most often left out, and that omission quietly caps how good a manipulation policy can get.

This post makes one argument. Proprioception is not a supporting actor in robot learning. For contact-rich manipulation it is load-bearing, and because passive video does not contain it, capturing proprioception well is one of the sharpest lines between data that trains a real policy and data that only looks useful.

What proprioception actually encodes

In a robot, proprioception is the set of internal signals that describe the body's own configuration and effort. It is not one number. It is a bundle.

• Joint positions: the angle of every actuated joint, usually from encoders, sampled fast.
• Joint velocities: how quickly each angle is changing, which the controller needs to stay stable.
• Joint torques or currents: the effort each motor applies, a cheap proxy for contact force.
• End-effector pose: where the hand sits in space, derived from joint angles through forward kinematics.

Together these tell the policy what the body is doing right now, independent of the scene. A vision system says the mug is over there. Proprioception says my hand is here, moving this fast, pushing this hard. A good manipulation policy needs both, fused at every control step.

Why manipulation collapses without it

Picture a peg entering a hole with a fraction of a millimeter of clearance. The instant the peg touches the rim, the camera can no longer see the contact. The fingers occlude it. The interesting physics happens exactly where vision goes dark.

What still works is the body's own report. A small rise in joint torque means the peg has caught an edge. A stall in joint velocity means motion stopped before the goal. Human hands solve these problems mostly by feel, not by sight, and robots have to do the same.

There is a second failure mode: drift. A policy that reads only pixels has no direct sense of its own motion, so small errors pile up with nothing to correct against. Proprioception closes that loop. It hands the policy a high-rate, low-latency signal about its own state that vision, at 30 Hz and often delayed, cannot match.

Vision tells a robot about the world. Proprioception tells it about itself. Take the second away and manipulation becomes a guess.

Sampling rate is part of the signal

Proprioception is worth little if it arrives slowly. Contact events happen in milliseconds. A slip, a collision, a clean mate, each shows up first as a spike in torque or a discontinuity in velocity, and if you sample at 30 Hz you miss it.

So capture rate matters as much as capture content. Joint states are usually logged far faster than camera frames, often at 240 Hz or higher, then aligned against the slower vision stream. The mismatch is deliberate. You want proprioception dense enough to catch the physics and vision dense enough to catch the scene, each at its own native rate. A dataset that logs everything at one slow rate throws away the part of the signal that made proprioception valuable. Passive first-person footage, the kind gathered in Ego4D, is rich in intent but carries no proprioception at all.

How we capture proprioception from humans

Human demonstration is the cheap, scalable source, but humans do not ship with joint encoders. So how do you get proprioception out of a person?

Two paths. The first is teleoperation: a human drives a real robot, and the robot's own encoders log ground-truth proprioception while the person supplies the intent. This is how datasets like DROID and RH20T get clean joint and force data. The catch is throughput and cost, since every hour of data needs a robot and an operator.

The second path reconstructs human proprioception directly, from wearables and vision. Inertial sensors, hand-tracking gloves, and multi-camera rigs estimate the joint angles of the body and hands, which are then retargeted onto a robot's kinematics. This is harder to get right, because a reconstructed joint angle is an estimate, not an encoder reading. But it scales in a way teleoperation never will, and it is where first-person capture earns its place.

Both paths feed the same downstream need: a state vector, time-aligned with vision and action, that a model such as NVIDIA Isaac GR00T can consume alongside camera frames and a language instruction. Corpora like Open X-Embodiment show both the value and the problem: breadth across many robots, but proprioceptive fields that are inconsistent, differently scaled, and sometimes missing.

The signal a robot trusts

Proprioception is unglamorous. It produces no striking visualization and no viral demo. But it is the signal a robot leans on when the lights go out and the fingers block the view, and that is exactly the moment manipulation succeeds or fails.

The teams building humanoid foundation models already know this. That is why their state vectors carry joint angles, velocities, and torques, not just pixels. The open question for anyone capturing training data is whether the pipeline preserves that signal at the rate physics demands, or quietly drops it and hopes vision will cover. It will not.