Keypoint Representations for Manipulation
Sparse keypoints give manipulation policies a category-level handle that generalizes to new objects where raw pixels and learned latents do not.
Two mugs sit on a shelf. One is a squat blue ceramic, the other a tall white travel cup. To a policy that reads raw pixels, these are two unrelated images with almost no overlap. To a policy that reads keypoints, they are the same three points: rim, handle root, base. One representation makes the mugs strangers. The other makes them siblings.
That gap is the whole argument for keypoints. A manipulation policy does not usually need the full pixel appearance of an object. It needs a handful of task-relevant locations, tracked reliably across instances and viewpoints. Get those points and a lot of generalization comes almost for free.
The cost is that someone, or something, has to define and find the points. That is where the real work, and the real data problem, lives.
Two mugs, three points
A keypoint representation replaces a dense image with a sparse set of labeled 3D locations that mean the same thing across every instance of a category. The handle keypoint is the handle whether the mug is blue, white, chipped, or half in shadow. Because the points are defined by function, not appearance, a policy built on them inherits a category-level invariance that a pixel policy has to learn from scratch.
This is an old idea in a new setting. Classical vision used keypoints for matching. Methods like kPAM and dense object descriptors brought the idea into manipulation, letting a policy specify a task as a relationship between semantic points: put the rim keypoint above the plate keypoint, align the peg keypoint with the hole keypoint.
Why sparse beats dense for generalization
The case for keypoints is a case about where generalization comes from. A raw-pixel policy has to disentangle what matters, the handle, from what does not, the color, the logo, the lighting, using only the training signal. With enough data it can. With realistic data it often does not, and it latches onto a spurious cue that happens to correlate with success.
Keypoints inject that disentanglement as a prior. By committing up front to "these locations are what matter", the representation throws away appearance the policy would otherwise overfit to. Work from groups like the Stanford IRIS lab on structured representations, and a steady stream of results in the current robotics literature, keep finding the same thing: sparse structure trades a little expressiveness for a lot of robustness.
Keypoints work because they throw away the part of the image the policy would otherwise overfit to.
Representation choices, side by side
| Representation | Generalizes to new instances | Sample efficiency | Fails when |
|---|---|---|---|
| Raw pixels | Poorly without huge data | Low | Appearance shifts, new backgrounds |
| Learned latent | Moderate | Moderate | Distribution shift, opaque failures |
| Object pose (6-DoF) | Well for rigid known objects | High | Objects deform or are unknown |
| Semantic keypoints | Well within a category | High | Occlusion, deformables, undefined points |
No row is a free win. Pose estimation is excellent when you have a CAD model and a rigid object, and useless for a towel. Keypoints shine within a category and struggle the moment a point is occluded or simply does not exist on a new object shape.
Where the keypoints come from
The representation is only as good as the correspondence behind it. To train a keypoint model you need many views of many instances with the points in agreement: this pixel here is the same semantic handle as that pixel there. Getting that correspondence is the data cost keypoints quietly impose.
Human demonstration helps in a specific way. When a person manipulates an object, their contact points are a strong prior on which locations are task-relevant, and their hand tracks those locations across the motion. Large multi-embodiment collections such as Open X-Embodiment supply the cross-instance variety that keypoint models need to avoid memorizing a single object, and physical-prior work from the NVIDIA GEAR Lab research group leans on structured intermediates of exactly this kind.
From points to actions
Keypoints are a state representation, not a policy. Something still has to turn "the peg keypoint is 3 cm from the hole keypoint" into motor commands. In practice this is where two design styles diverge. One feeds the keypoints to a learned policy as a compact observation, letting the network map points to actions. The other writes the task as an explicit geometric objective, drive these points into that configuration, and hands it to an optimizer.
The learned route is more flexible and copes with contact and dynamics that pure geometry ignores. The optimization route is more interpretable and needs almost no task-specific data once the keypoints exist. Most working systems blend them, using keypoints to define the goal and a learned policy to close the last centimeters where contact gets messy. Either way, the representation earns its keep by making the downstream problem smaller.
The limits, stated plainly
Keypoints are not a universal state. Deformable objects break the assumption that a fixed set of points describes the object; a folded shirt has no stable "corner three". Heavy occlusion hides the very points the policy depends on. And someone still has to decide which points matter for a task, a choice that does not automate cleanly.
The honest position is that keypoints are a powerful default for rigid and articulated objects within known categories, and a poor fit for the deformable long tail. Knowing which regime you are in is half the battle.
Raw pixels tell a policy everything about an object, which is precisely the problem. Most of it is noise for the task at hand. Keypoints are a bet that a handful of well-chosen locations carry the signal and drop the rest, and within the right regime the bet pays off in generalization you did not have to buy with data. The catch is that the points have to come from somewhere. That somewhere, more and more, is human hands doing the task, captured so the structure survives.