Camera Calibration: The Step That Decides Usability

Every mug the robot reaches for, it stops three centimeters short. Same error, every time. The policy is not broken. The camera was never calibrated properly, and a systematic three-centimeter bias baked into the training data taught the arm to aim three centimeters wrong. No amount of extra demonstrations fixes it, because every new demonstration carries the same lie.

Calibration is the least discussed step in a capture pipeline and one of the few that can silently invalidate everything downstream. It is unglamorous. It involves waving a checkerboard around and staring at reprojection numbers. Skip it, or do it carelessly, and you can collect a thousand hours of beautifully recorded, geometrically worthless data.

Intrinsics, extrinsics, distortion: three things that must be right

A camera turns a 3D world into a 2D grid of pixels. Recovering geometry from that grid needs three sets of numbers, and each fails in its own way.

Intrinsics describe the camera itself: focal length, principal point, sensor scale. They map a pixel to a ray leaving the lens. Get them wrong and every depth estimate and every projected position is scaled or shifted.

Extrinsics describe where the camera sits relative to the robot or the world: its position and orientation. They are what lets you say a pixel corresponds to a point in the robot base frame. A one-degree error in orientation becomes centimeters of error at arm's length.

Distortion describes how the lens bends light. Straight lines near the edge of a wide lens bow inward. Uncorrected, a gripper tracked across the frame appears to curve along a path it never took.

Solving for these numbers is the calibration procedure itself. Show the camera a known pattern, a checkerboard or a grid of markers, from many angles. An optimizer then finds the intrinsics and distortion that best explain where the pattern corners land in each image. The residual it leaves behind, the reprojection error, is the honest measure of how well the model fits. Sub-pixel is the target. A few pixels means something is wrong.

A demonstration is only as trustworthy as the geometry that turned its pixels into positions. Miscalibrate the camera and every label in the dataset inherits the error.

Why calibration decides whether a demonstration is even usable

Think about what a manipulation demonstration actually is. It is a mapping from what the camera saw to what the hand did. If the geometry connecting pixels to 3D space is wrong, then the recorded relationship between perception and action is wrong too. The policy dutifully learns a broken correspondence.

This is worse than random noise. A miscalibration is systematic. It biases every sample in the same direction, so the model does not average it away, it absorbs it as truth. Datasets built on standardized rigs like DROID put effort into consistent calibration precisely so that data pooled across many stations stays geometrically comparable.

Egocentric capture makes the stakes plain. When the camera rides on a human head, it moves constantly, and the hand it is watching is close, where small angular errors translate into large positional ones. Reading a fingertip position to within a centimeter from a moving first-person view leaves almost no calibration budget to spare.

The bottom two rows matter more than teams expect. A wrist camera and a clock offset both turn a static calibration problem into a moving one, and moving errors are the ones people forget to check.

Multi-camera and eye-in-hand: where it gets hard

One fixed camera is the easy case. Real rigs are messier. Multi-view capture, the kind Ego-Exo4D uses to see one action from egocentric and exocentric angles at once, needs every camera calibrated into a shared frame, or the views will not agree on where anything is. The value of many viewpoints depends entirely on their extrinsics being mutually consistent.

Wrist-mounted cameras are harder still. The camera moves with the hand, so its relationship to the gripper must be solved by hand-eye calibration, and that relationship has to hold as the joint angles change. Datasets that combine fixed and moving views, like RH20T, carry this burden for every camera they add. When corpora are merged, as in Open X-Embodiment, inconsistent calibration conventions between source labs become one more thing a unified policy has to tolerate.

Calibration drift and the discipline of re-checking

Here is the trap that catches careful teams. Calibration is not permanent. A camera gets bumped during a reset. A rig warms up and the mount expands by a fraction of a millimeter. Someone refocuses a lens and quietly changes the intrinsics. None of these throw an error. The data keeps recording, now subtly wrong, and the defect only surfaces weeks later as a policy that mysteriously underperforms.

The remedy is process, not cleverness. Re-check calibration on a schedule and after any physical disturbance. Track reprojection error and flag a rig when it climbs past a threshold, ideally well below a single pixel for close manipulation. Store the calibration state alongside every recording, so a trajectory can be traced to the exact parameters in force when it was captured. Research groups such as the Stanford IRIS lab treat this reproducibility as part of the science, not an afterthought.

Calibration will never be the exciting part of a capture operation. It photographs badly and demos worse. But it sits underneath every other claim a dataset makes, because it decides whether a pixel corresponds to a place in the world or merely to a guess. Get it right and the rest of the pipeline has a foundation. Get it wrong and you are training robots on a map that does not match the territory.