RGB vs Depth Cameras for Robots

Point a depth camera at three things a home robot touches every day: a black leather wallet, a glass of water, and a chrome tap. Watch what returns. The wallet reads as a void, the water surface boils with noise, and the tap smears phantom geometry across half the frame. Depth sensors fail on exactly the materials that fill a kitchen.

That is the uncomfortable subtext behind one of the most routine decisions in a robot data stack. RGB, depth, or both? The question sounds like a hardware detail. It is really about bandwidth, failure modes, and what a policy can learn from the pixels you actually record.

Depth is not free. It costs bandwidth, calibration, power, and money, and it degrades in the cases that matter most. RGB is cheap and universal, but it hands the geometry problem to the network. Knowing when each choice pays off is worth real money once you multiply it across thousands of hours of capture.

What a depth camera actually gives you

A depth camera returns a distance for each pixel. Instead of color, you get metric geometry: this surface is 42 centimeters away, that edge is 3 centimeters closer. For a grasp planner, that is gold. You can segment an object from its background by range alone, estimate its pose, and check whether a gripper will clear the shelf above it.

There are three common ways to get it. Stereo pairs triangulate from two RGB lenses, the approach behind large real-robot sets like the DROID dataset. Structured light projects a known pattern and reads its distortion. Time-of-flight measures how long a light pulse takes to return. Each has a signature weakness. Stereo needs texture and struggles on blank walls. Structured light washes out in sunlight. Time-of-flight scatters on shiny and transparent surfaces.

So the promise of per-pixel geometry comes with an asterisk. Depth is most reliable on matte, opaque, well-lit, textured objects, and least reliable on glass, chrome, dark cloth, thin wires, and anything outdoors at noon. Those are not edge cases in a real home. They are Tuesday.

What RGB alone can already do

A decade ago, throwing away depth looked reckless. Not anymore. The dominant robot foundation models are trained largely on RGB video, because that is what exists at scale. Most of the trajectories aggregated in Open X-Embodiment are plain color streams, and policies built on them still learn to grasp, push, and place.

Two shifts drove this. First, large vision backbones learned to infer geometry implicitly from monocular cues: shading, occlusion, perspective, motion parallax. Second, the field moved toward end-to-end policies that never build an explicit 3D map. A model like the ones targeted by NVIDIA Isaac GR00T maps images and language straight to actions, and it recovers a surprising amount of the structure a depth sensor would have measured.

RGB still has one hard limit: scale ambiguity. A single color image cannot tell a real mug from a dollhouse mug at the same viewpoint. Motion, a second camera, or a known gripper in frame resolves it. But if your task depends on absolute millimeters, RGB alone is a gamble.

There is a subtler cost to leaning on RGB. When the network infers geometry, that inference is only as good as the training distribution. Show it a transparent bottle it never saw, and the guess degrades in ways a depth reading would not. RGB shifts the failure from the sensor to the data. That is a reasonable trade, but only if your data actually covers the hard cases.

When depth earns its bandwidth

Depth stops being optional in a few concrete regimes.

• Cluttered bin picking. When objects overlap and occlude, range data separates them far more cleanly than color does. This is why industrial pick systems still lean on depth.
• Novel objects with no prior. If the policy has never seen an object and has no language hint, geometry is the only reliable handle on where to grasp.
• Collision-sensitive motion. Reaching into a shelf, or between two fragile items, you want measured clearance, not a guess.
• Precise placement. Setting a lid on a pot, or a peg near a hole, rewards a few real millimeters over an inferred estimate.

Work surveyed on the Berkeley BAIR blog keeps returning to the same pattern. Depth helps most where contact is imminent and error is expensive. Away from contact, during the long reach across a room, it mostly adds cost.

There is nuance even here. Transparent and reflective objects break depth too, so the exact materials that make RGB struggle can also defeat a depth sensor. Recent work pairs the two, using color to repair the holes that time-of-flight leaves on glass. The lesson is not that depth is a cure. It is that the two signals fail on different inputs, and together they cover more than either alone.

Depth is not a better camera. It is an expensive insurance policy that pays out only in the last few centimeters before contact.

What this means for the data you capture

Here is the asymmetry that should drive the decision. If you record depth and later decide you do not need it, you can drop it in an afternoon. If you record RGB only and later need metric geometry, you cannot recover it. The information was never there.

That argues for capturing more than the current model consumes, as long as the streams are properly registered and time-synced. Aligned RGB plus depth, with intrinsics and extrinsics stored alongside, lets a downstream team choose per task. Ship only color and you have quietly capped what buyers can ever train.

Storage is the counterargument, and it is real. Depth roughly doubles the bytes per camera, and at facility scale that is a line item, not a rounding error. But storage is cheap and getting cheaper, while a lost recording is gone for good. Compress the depth, prune it later, archive it cold, but capture it while the moment still exists.

So RGB or depth is the wrong framing. The real question is where in the task the geometry has to be measured rather than guessed, and whether you can afford to be wrong there. For a slow reach across an empty table, RGB carries the load. For the final centimeters into a cluttered shelf, depth still earns its bandwidth. A serious dataset captures both, keeps them aligned, and lets the model decide.