Representation Learning for Manipulation
Two policies, same demos, different features, opposite results. How representation learning quietly decides whether a manipulation policy generalizes.
Take two teams, hand them the same 500 demonstrations of a robot picking up a mug, and let them train the same policy network. One team feeds the policy raw camera pixels. The other feeds it features from a vision encoder pretrained on millions of internet images. On the training mugs, both look fine. Move to a mug the robot has never seen, under a lamp it has never seen, and the gap opens: one policy finds the handle, the other paws at bare table.
Nothing changed but the representation. Same data, same network, same reward signal. The features a policy is allowed to see decide most of what happens next. This is the quiet lever in manipulation, and it is where a great deal of real success and failure is actually settled.
So it is worth being precise about what a representation is, where it comes from, and why the answer is mostly a question about data.
The feature vector is the policy's whole world
A policy maps observation to action. Between those two ends sits an encoder that turns pixels, depth, joint angles, and force readings into a compact vector. That vector is everything the policy knows. If the encoder discards the position of the mug's handle, no amount of downstream training will bring it back.
Strong manipulation features tend to share a few traits. They are spatially grounded, so the policy reasons about where things sit in three dimensions rather than what merely appears in frame. They stay temporally consistent, so a slow reach reads as one motion and not ten unrelated snapshots. They shrug off distractors: a cluttered counter, a drifting shadow, a lamp switched on. And for contact-rich tasks they carry force, because you cannot see a grasp beginning to slip.
Miss any one of these and the failure is specific. Drop 3D and the robot misjudges depth. Drop temporal structure and it stutters. Drop force and it crushes the paper cup it was told to lift gently.
Where manipulation features come from
There is no single source of features, and the choice is a real engineering decision with real tradeoffs. Broadly, five options are in play, and most modern systems blend them.
| Source | What it is | Strength | Main limitation |
|---|---|---|---|
| End-to-end from scratch | Encoder trained jointly with the policy on robot demos | Maximally task-relevant features | Data-hungry, overfits, transfers poorly |
| Supervised web images | ImageNet-style classification backbone | Cheap, ubiquitous, well understood | Object-label bias, weak 3D and no contact |
| Language-aligned | Image-text encoder such as CLIP | Semantic, easy to condition on instructions | Coarse spatially, blind to force |
| Self-supervised, in-the-wild | Encoders like DINO, R3M, VC-1 learned without labels | Strong general priors, no annotation cost | Domain gap to the robot's embodiment |
| Multimodal robot capture | Vision fused with proprioception, force, and touch | Contact-aware, close to the task | Needs instrumented, synchronized capture |
The interesting rows are the last two. Self-supervised encoders trained on ordinary video, several of them out of Berkeley BAIR blog and its collaborators, gave the field its first widely reused visual priors for control. They work because manipulation shares structure across scenes, and that structure survives being learned without labels.
Frozen, fine-tuned, or learned from scratch
Once you have a pretrained encoder, the next fork is whether to freeze it. Freezing is cheap and stable: the features stay put while the policy learns on top. It also caps how well the system can adapt, because a frozen encoder cannot notice a cue it was never pretrained to represent.
Fine-tuning the encoder on robot data closes that gap, at the cost of more demonstrations and a real risk of overfitting to the lab. Large recent VLA stacks, including work around NVIDIA GEAR Lab research and generalist policies described by Physical Intelligence blog, lean toward fine-tuning a strong backbone rather than starting cold. The pattern is consistent: pretrain broad, adapt narrow.
A manipulation policy is only ever as smart as the features it is allowed to see. Everything after the encoder is arithmetic on those features.
Why pixels alone hit a ceiling
Web-pretrained vision has a hard limit for manipulation: it was optimized to answer what is in the image, not how to act on it. A model can name a mug flawlessly and still have no idea how a hand pre-shapes before touching it. That knowledge lives in interaction, not in captions.
Two responses have held up. The first is object-centric and keypoint representations, sparse points on an object that a policy can track and generalize across instances, an approach pushed hard at Stanford IRIS lab. Keypoints throw away most of the image and keep the part that matters, which is exactly why they transfer. The second is fusing non-visual channels: proprioception tells the robot where its own joints are, and force-torque tells it what the world is pushing back with. Neither is visible in a photograph.
Representations inherit their training data
Here is the part that gets underweighted. A representation is a compression of its pretraining distribution, so it inherits that distribution's blind spots. Train on captioned web images and you learn object identity. Train on first-person video of people doing things with their hands and you begin to learn affordances, contact, and the choreography of a grasp.
This is why egocentric corpora like Ego4D and cross-embodiment collections like Open X-Embodiment matter beyond their raw size. They shift the pretraining distribution toward the thing the robot actually has to do. The features that come out are closer to manipulation on the first day, before a single robot demonstration is collected.
The effect is measurable in transfer. A backbone pretrained on hand-object video tends to need fewer robot demonstrations to reach a given success rate than one pretrained on object-classification images, because it starts already sensitive to grasp points and contact. The pretraining set is a prior, and a good prior is worth demonstrations you never have to collect.
The practical takeaway
If a manipulation policy is failing to generalize, the encoder is the first place to look, not the last. Ask what the features can and cannot represent. Ask what data taught them. Swapping a stronger, better-matched representation in often does more than another thousand demonstrations on the same backbone. The features are the foundation, and foundations are hard to fix once the house is up.