Motion Capture Suit vs Vision-Only Capture

A marker-based motion capture suit can pin a fingertip to well under a millimeter, sixty times a second, in a room ringed with infrared cameras. It also takes twenty minutes to put on, needs a calibrated studio, and makes the wearer move like someone who is wearing forty reflective markers. Vision-only capture reads the same motion from ordinary cameras, with nothing on the body, at a fraction of the accuracy. That gap is the whole debate.

For human demonstration data, this is the first fork in the road. Do you instrument the person, or infer everything from pixels? The answer shapes cost, scale, realism, and, above all, how well the hands come through.

What an instrumented suit buys you

Put markers or inertial sensors on a body and you get ground truth. Optical systems triangulate marker positions to sub-millimeter accuracy. Inertial suits fuse gyroscopes and accelerometers to report joint angles directly, even when a limb is hidden from every camera. The output is a clean, high-rate skeleton: joint angles you can retarget onto a robot without guessing.

Rate matters here. A good suit streams at 240 Hz, fast enough to catch the quick reversals of real manipulation, which you can later downsample to a 30 Hz output for training. Multi-view efforts like Ego-Exo4D pair egocentric video with calibrated external cameras precisely to recover this kind of high-fidelity pose.

The bill comes due elsewhere. Suits take time to don and calibrate. Optical rigs tie you to a studio. Inertial systems drift and hate magnetic clutter. And every marker is a small tax on natural behavior, because a person who is aware of the suit does not move quite like a person washing dishes at home.

Fidelity is not only spatial. A suit reports a full joint state, position and often velocity, at every timestep, which is exactly the proprioceptive signal a policy needs and a camera cannot see. Vision gives you where the hand appears to be. A suit gives you what the joints are actually doing, including the small preloads and tremors that never show up as visible motion.

What markerless vision-only capture buys you

Vision-only capture flips every tradeoff. There is nothing to wear. You point cameras, record, and estimate pose from the images afterward. That makes it cheap, fast to deploy, and usable in real kitchens and workshops where a studio will never fit. The largest egocentric human dataset, Ego4D, is vision-only for exactly this reason: it scales to thousands of hours in the wild.

The behavior you capture is also more natural. Nobody is thinking about their markers, because there are none. People act closer to how they actually act, which is the entire point of learning from human demonstration.

What you give up is fidelity, especially under occlusion. A hand that turns away from the camera becomes a guess. Depth is ambiguous from one view. Fast motion blurs. Modern pose models are strong, but they still infer where a suit would measure, and inference fails quietly in exactly the cluttered, close-contact moments manipulation cares about most.

The gap is closing, slowly. Multi-camera rigs and learned priors have pushed markerless hand tracking a long way, and for coarse whole-body motion it is often good enough. But error is not uniform. It concentrates in the fingers, under occlusion, and during fast contact, which are precisely the moments a manipulation policy is trying to learn.

The hidden variable is the hands

Whole-body pose is the easy part. Manipulation lives in the fingers, and fingers are where the two approaches diverge hardest. A gloved suit records each finger joint directly, even mid-grasp with the hand wrapped around an object. Vision has to reconstruct twenty-plus finger degrees of freedom from an image where half of them are self-occluded by the object being held.

This is why a dataset can look rich at the body scale and still be thin where it counts. If the fingers are guessed, contact-rich skills inherit the error.

Dexterous hardware only sharpens the point. As robot hands gain fingers and degrees of freedom, the fidelity of the finger data you trained on becomes the ceiling on what the hand can learn to do.

For manipulation data, the accuracy that matters is not where the wrist is; it is what each finger is doing at the moment of contact.

Building a demonstration corpus without choosing sides

Suit versus vision is a false binary. Serious capture blends them. Markerless collection buys breadth: many people, many homes, many hours, cheaply. An instrumented core buys depth: a smaller set of high-fidelity, fully observed demonstrations that anchor and calibrate the rest. Cross-register the two and the cheap data inherits some of the credibility of the expensive data.

Retargeting human motion onto a humanoid, a focus of the NVIDIA GEAR Lab research, only works if the source joints were measured accurately in the first place. Whatever the mix, the metadata decides usability. Every trajectory needs its capture method, calibration, and rate recorded, so a downstream team knows which joints are measured and which are inferred. Method surveys on arXiv Robotics (cs.RO) increasingly report this provenance as a first-class field, not a footnote.

There is a consent and privacy dimension that favors deliberate capture too. A studio session with informed participants is easier to govern than video scraped from the open web, and the resulting metadata, who was recorded, when, and under what agreement, travels with the data.

So neither the suit nor the camera is the right answer on its own. The suit knows exactly what the hand did and struggles to scale. Vision scales to the whole messy world and guesses at the fingers. The corpus that will actually train dexterous policies is the one that uses cheap vision for coverage, an instrumented core for truth, and careful metadata to tell a buyer which is which.