Why dexterous hands need richer manipulation data

Set a coin flat on a table and pick it up with your fingertips. Now try it with kitchen tongs. The tongs manage a pencil or a mug, but the coin defeats them, because lifting something thin and flat off a flat surface needs a fingernail sliding under an edge, a pinch, then a small roll of the object into the palm. That gap, between what tongs can do and what fingers can do, is the gap between the robot hands most training data describes and the robot hands the field actually wants to build.

A two-finger parallel gripper doing pick-and-place is a largely solved data problem. There are big open corpora of it, and a modern policy learns it from a few hundred demonstrations. A five-finger hand reorienting a part in mid-air, threading a cap, or tying a lace is a different problem by orders of magnitude, and the data to teach it barely exists. The reason is not that nobody has tried. It is that dexterity raises the data bar along several axes at once.

This piece is about why. Not that dexterous is harder, which everyone agrees on, but specifically what dexterity demands from a training record that a gripper never did: far more degrees of freedom to supervise, contact events that are faster and more numerous, in-hand slip and regrasp that a single camera cannot resolve, and finger coordination with no cheap shortcut.

The degrees-of-freedom gap

Start with the count. A parallel-jaw gripper has one actuated degree of freedom: it opens and it closes. Everything else about where it goes is the arm's job. Bolt a five-finger hand onto the same arm and you add, depending on the design, somewhere on the order of sixteen to twenty-four actuated joints, all moving together. The human hand that a robot hand imitates carries over twenty degrees of freedom of its own.

This is not a linear bump in difficulty. A demonstration has to supply a supervision target for every controlled joint at every timestep. For a gripper, that target is close to a single scalar: how open. For a dexterous hand it is a high-dimensional vector that has to be internally consistent, because the fingers share tendons, touch the same object, and constrain each other. You cannot label the thumb without reference to what the other four fingers are doing. The action space the policy must learn to produce jumps from one channel to roughly twenty coordinated channels.

A gripper demonstration answers one question at each instant: how wide. A dexterous demonstration answers twenty at once, and they only make sense together.

In-hand manipulation is a different animal

Pick-and-place, even with a fancy hand, is mostly transport: grasp, move, release. The grasp is static once it is formed. Dexterity, in the sense that matters, is what happens when the object keeps moving after you have grabbed it. Reorienting a bolt to find its thread. Walking a pen across the fingers to bring the tip forward. Regrasping a tool that landed in the hand at the wrong angle. Roboticists call the finger-by-finger relay that makes this possible finger gaiting, and it has no equivalent in a gripper's vocabulary.

In-hand manipulation breaks the assumptions the cheap data was built on. Contact is no longer a single make-and-break at the fingertips; it is a rolling sequence of makes and breaks spread across five fingers, each briefly load-bearing and then released as another takes over. Slip stops being a failure to avoid and becomes a controlled tool: you let the object slide a measured amount, then arrest it. OpenAI's Dactyl work reoriented a block and later a Rubik's cube in a single hand, but it leaned almost entirely on simulation and domain randomization, precisely because a real corpus of that behavior did not exist to learn from.

Why one wrist camera cannot see it

The standard sensor package that made gripper data cheap, a wrist camera plus joint encoders, quietly stops being enough here. The problem is self-occlusion. During in-hand manipulation the object sits in the middle of the hand, surrounded by the very fingers doing the work. A wrist camera looks down the back of the hand at a cluster of fingers that hide both the object and each other. The most information-dense region of the scene is the one most reliably blocked.

Extra external cameras help, but they hit the same wall from outside: at any instant most fingertips are occluded by the hand's own geometry. That is why dexterous capture cannot lean on vision to recover the state after the fact. The finger joint angles have to be measured directly, at the source, by the glove or the hand's own encoders, because there is no camera angle from which all of them are visible at once. Groups pushing on learned dexterity, such as Stanford's IRIS lab and the wider in-hand manipulation literature on arXiv, keep running into the same constraint: the ground truth for a hand is not reliably recoverable from pixels.

What a dexterous demonstration must carry

Put the requirements side by side and the jump is stark. The same task verb, pick it up, hides a step change in what the recording has to contain.

Two rows deserve emphasis. Frame rate: a gripper's single contact transition can be caught by a fast tactile channel without much fuss, but finger gaiting packs several contact transitions into the time a gripper needs for one, so the force and tactile streams have to run at hundreds of hertz or the regrasp becomes a blur between samples. And per-fingertip force: a single wrist force-torque sensor reports the net wrench on the whole hand, which says nothing about which finger is slipping. Dexterity needs force resolved per contact, which means tactile sensing on each fingertip, calibrated to physical units.

Why this data barely exists at scale

Everything above makes dexterous demonstrations expensive to produce, and that expense shows up as absence. The large open manipulation corpora are overwhelmingly gripper data, because that is what most research robots carry and what teleoperation rigs cheaply produce. Datasets that pair vision with real force and tactile readings, like RH20T, are the exception rather than the rule, and even they lean toward simpler end effectors. High-DOF, per-finger, force-instrumented in-hand data is the thinnest slice of an already thin supply.

The field's current answers are revealing. NVIDIA's Isaac GR00T program blends real robot data, human video, and large volumes of simulation precisely because real dexterous demonstrations are too scarce to train on alone, and its GEAR lab has pushed on generating dexterous demonstrations synthetically to multiply the little real data that exists. Physical Intelligence has shown generalist policies on genuinely dexterous, deformable tasks, and the through-line of that work is that the hard part was assembling demonstrations of the behavior in the first place. When the standard move is to manufacture data in simulation, it is a tell that the real thing is missing.

So the honest way to read a clip of a robot hand tying a knot is not that the model got smart. It is that someone solved the data problem for that one behavior, usually with a long detour through simulation. Grippers taught the field that manipulation could be learned from a few hundred demonstrations. Hands are teaching it the fine print: that the number was always a function of how many things the end effector could do, and a hand can do a great many more. Closing the gap is less about a cleverer policy than about going out and recording, joint by joint and finger by finger, the dexterity that no gripper corpus ever held.