Retargeting: mapping a human demonstration onto a robot body

A person lifts a mug by hooking one finger through the handle, thumb resting on the rim. It costs them no thought. Now hand that same clip to a robot whose hand is a two-finger parallel gripper. There is no finger to hook, no spare digit, no rim-steadying thumb. The demonstration is flawless and unusable at the same instant, because it is written in a body the robot does not have.

This is the quiet tax on every promise that cheap human video will feed robot learning. A camera on a line cook's forehead captures hundreds of grasps an hour, but not one of them arrives in a form a gripper can run. The joint angles belong to a human hand. The reach belongs to a human arm. The timing belongs to a human nervous system. Between the footage and an executable robot action sits a translation step, and it has a name: retargeting.

Retargeting maps a demonstration from the body that produced it onto the body that has to reproduce it. Do it well and a mountain of human video turns into training signal. Do it badly and you have taught a robot to chase a motion it can never physically make. It is worth walking through as a pipeline, because every stage in it leaks, and the leaks decide how much of that cheap footage was ever worth collecting.

The problem underneath

Roboticists call the core issue the correspondence problem: two agents with different bodies cannot share an action, only an intent. A human hand carries more than twenty degrees of freedom. A parallel-jaw gripper has one. A three-fingered robot hand has a handful. The demonstration lives in the first space and the robot lives in the second, and nothing carries one into the other for free.

What survives the crossing is intent: the object approached, the way it was grasped, the goal state reached. What does not survive is the literal motion. So retargeting is not a filter you run once. It is a chain of estimation and optimization, and error compounds down the chain. Understanding where it compounds is the difference between using human video well and fooling yourself with it.

Stage one: recover what the human did

Before anything can be mapped onto a robot, the human motion has to be reconstructed from pixels. An egocentric clip is not a list of joint angles; it is RGB, sometimes depth, sometimes a wrist view. The first stage estimates 3D hand pose, body pose, and object pose per frame: fingertip positions, wrist trajectory, the object's six-degree-of-freedom pose, and ideally the moments of contact.

This stage is already lossy. Monocular pose estimation guesses depth it cannot see, fingers occlude each other during a grasp, and the exact instant of contact is inferred from geometry rather than felt. Paired-camera datasets exist partly to fight this. Ego-Exo4D records the same skilled activity from synchronized first- and third-person cameras, which gives pose reconstruction more than one line of sight and a far better shot at accurate 3D keypoints. Whatever error survives this stage, every later stage inherits.

Stage two: solve for the robot's body

Now the recovered human keypoints become a target the robot has to hit, and this is where inverse kinematics enters. IK solves for the joint angles that place the robot's end effector, its gripper or hand, where the demonstration says it should go. In practice retargeting picks what to match, because it usually cannot match everything.

Two strategies dominate. End-effector retargeting tracks where the wrist and gripper travel and solves IK to follow that path, treating the hand as a single functional unit. Keypoint retargeting goes finer: it minimizes the distance between human fingertip vectors and the robot's fingertip vectors, so a pinch stays a pinch even on a hand with fewer fingers. Dexterous-hand methods in the cs.RO literature lean on this vector-matching idea to drive multi-fingered hands from human video.

The subtler half is grasp and contact matching. What a demonstration is really about is the effect on the object, not the shape of the hand that caused it. Functional retargeting tries to preserve that effect: reproduce a stable grasp with the same approach and the same contact geometry, even if the robot achieves it with two pads where the human used five fingers. Match the outcome on the object, not the silhouette of the hand, and more demonstrations survive.

Where retargeting leaks

Three mismatches force the hardest compromises, and each one costs you demonstrations.

Finger count. Mapping a five-finger hand onto three fingers or two pads is many-to-few. A tripod grasp has a plausible three-finger image and a rough two-finger one. An in-hand regrasp, where the fingers walk an object into a new pose, has no image at all on a parallel gripper. Those strategies do not retarget; they drop out.

Workspace. The robot's arm has its own reach, its own joint limits, its own self-collisions. A human reach that swings behind a shelf may have no valid IK solution on the robot, so the path has to be scaled, warped, or rejected. Rescaling the workspace keeps the shape of a motion while moving it into reachable space, at the cost of exactness.

Speed and dynamics. Humans move faster than most robots are rated to, and flick the wrist in ways a heavier arm cannot. A demonstration that exceeds the robot's velocity or torque limits must be slowed or smoothed, which can quietly change a task whose success depended on momentum.

Retargeting can re-solve geometry it was given, but it cannot recover a signal that was never captured. A grasp can be re-derived from fingertip positions. The three newtons that kept the glass from slipping, if no sensor felt them, are simply gone.

Capture quality sets the ceiling

Here is the conclusion the pipeline forces. Retargeting fidelity caps how useful cheap human video is, and capture fidelity caps retargeting. You cannot remap a contact force you did not measure, or solve accurate IK against a 3D hand pose you estimated badly. The lossiest pipelines are the ones fed the thinnest data: monocular RGB, no depth, no force, pose guessed and contact inferred.

The fix is to move signal upstream, to capture time, where it is cheap to record and impossible to reconstruct later. Multi-view rigs pin down 3D hand and object pose so stage one stops guessing. Depth and wrist cameras tighten the geometry IK has to solve against. Tactile and force sensing turns grip pressure from a lost channel into a retargetable one. This is the logic behind paired human-and-robot capture in datasets like DROID, which give the retargeting step a supervised bridge rather than an open-loop guess. NVIDIA's Isaac GR00T stack and groups like Physical Intelligence lean on human demonstration precisely because it scales, but that leverage only cashes out to the degree the demonstration was captured richly enough to retarget with low loss.

So retargeting is not a preprocessing footnote between video and policy. It is the valve between the cheapest data source in robotics and an action a robot can actually run, and how far that valve opens is decided long before the solver runs, by what the camera and the sensors on the hand were able to catch. Capture thin, and you retarget a guess. Capture rich, and the human demonstration survives the trip into a body that was never built to make it.