Affordance Learning for Robots, from Human Hands

Reach for a lever door handle and your hand pre-shapes to push down before a finger touches the metal. You did not run a physics solver. You read the handle. It afforded pushing, and your motor system committed to that before contact.

This is the quiet skill behind almost every manipulation a person does. We do not perceive objects as geometry to be analyzed. We perceive them as sets of action possibilities: a mug affords lifting by the handle and drinking from the rim, a drawer affords pulling, a switch affords flicking. Roboticists borrow a word for this from perception research: affordances.

Teaching a robot affordances, rather than raw pixel-to-torque maps, is one of the more data-efficient bets in the field. And the best teacher is a human hand doing the task.

What a door handle tells you before you touch it

An affordance is a pairing between an object and an action it supports for a given agent. The handle affords pushing for a hand and, differently, for a two-finger gripper. That agent-relative part matters. A grasp that works for five fingers may be impossible for a parallel jaw, so an affordance learned from human hands has to be retargeted, not copied.

The payoff is compression. Instead of learning a separate policy for every object, a robot that predicts "where can I act, and how" reduces a cluttered scene to a short list of candidate actions. Perception stops being a labeling problem and starts being an action problem.

Affordances as a compression layer

Consider the alternative. An end-to-end policy maps pixels straight to motor commands. It can work, but it has to rediscover, from scratch and for every object, that handles are for holding and rims are for pouring. That is a staggering amount of data to spend on facts a human toddler already has.

Affordances sit in between. They are a mid-level representation: more abstract than pixels, more concrete than a language label. A model that outputs contact points and approach directions for a novel object can hand those to a low-level controller and skip relearning control from zero. Research groups including Berkeley's Berkeley BAIR blog have pushed on learning this kind of interaction prior from video, and the physical-prior work coming out of the NVIDIA GEAR Lab research group aims at the same target.

An affordance is the compression layer between seeing an object and knowing how to act on it, and human hands are the cleanest annotation of it we have.

Where affordance supervision comes from

Not all affordance data is equal. The source decides how well the learned affordances transfer, and at what cost.

The last row is the interesting one. When a person picks up a hundred different objects across a day, they demonstrate a hundred functional grasps, each already solved for stability and task. First-person datasets like Ego4D and its multi-view successor Ego-Exo4D capture exactly this: hands interacting with everyday objects at scale, from the viewpoint the robot will eventually use.

Human hands are affordance annotations

The trick is that a human hand touching an object labels the affordance for free. The contact points are where the fingers land. The approach direction is the path the hand took in. The task context is whatever the person did next. No annotator drew a mask; the demonstration is the label.

There is a catch, and it is the retargeting gap. A robot with two fingers cannot reproduce a five-finger power grasp, so the learned affordance has to be mapped to the robot's own kinematics. Contact regions transfer better than exact finger poses. Approach directions transfer well. Precise in-hand manipulation transfers worst. A capture pipeline that records where and how the hand contacted the object, not just an RGB clip, is what makes that mapping possible.

Affordances change with the task

A quiet failure mode is treating affordances as fixed properties of an object. They are not. The same knife affords a blade grasp for cutting and a handle grasp for passing it to someone safely. A screwdriver affords a precision grip for a small screw and a power grip for a stuck one. The action possibility that matters depends on what the agent intends to do next, which means affordance data has to carry task context, not just object identity.

This is why passive object detection falls short. Knowing a cup is present does not tell a robot whether to grab it for drinking, for washing, or for moving it out of the way, and each of those wants a different contact. Human demonstration carries that intent implicitly. The way a person grasps an object already encodes what they are about to do with it. Strip the task and you strip half the signal.

Dynamic affordances also raise the capture bar. A single frame cannot show that a drawer slides rather than swings, or that a bottle cap unscrews rather than pops off. You need the short window of motion around the interaction, which is exactly what continuous first-person capture preserves and a still image throws away.

The door handle never sends you a spec sheet. It just tells you, at a glance, what it is for, and your hand believes it. Getting robots to that same glance is less about bigger models and more about the right data: human hands, real contact, captured so the action possibility survives the jump to a robot body. Do that and a novel object stops being a new problem. It becomes a familiar set of affordances the robot has seen a thousand times before.