Mobile manipulation: data for robots that move and grab
Most robot data assumes an arm bolted to a table. Robots that walk to the object and then grab it need data that couples navigation with manipulation.
A robot in a warehouse walks to a shelf, stops, reaches into a tote, lifts a package, turns, and carries it to a conveyor. Every part of that looks routine until you ask where the training data came from. Almost all of it was recorded by an arm that cannot walk anywhere: bolted to a bench, staring at the same patch of table, repeating pick-and-place into the same bin for a thousand episodes.
This is the quiet mismatch inside mobile manipulation. The field has built an enormous supply of demonstration data on the assumption that the base never moves. The moment the base does move, walking, rolling, crouching, the problem changes shape, and much of that data stops being a clean fit. Mobile manipulation is not fixed-base manipulation with a drivetrain added underneath. It is a different data problem.
The difference comes down to one thing: a mobile robot has to get to the object before it can act on it, and how it got there decides whether the action is even possible. Capturing that coupling, navigation and manipulation as one continuous behavior rather than two, is harder than it looks, and it is where egocentric capture from a moving body earns its keep.
Most manipulation data assumes a table
Open a typical manipulation dataset and the base is furniture. The arm is mounted to a fixed frame, the workspace is a tabletop, the cameras watch from set positions, and the object sits somewhere the arm can already reach. That assumption is not an accident. It makes everything downstream simpler: the world frame and the robot frame are effectively the same, the reachable set is fixed, and an episode is short because the interesting part starts the instant the demonstration begins.
Open X-Embodiment pooled well over a million trajectories from more than twenty robot types, and it is a landmark in cross-embodiment learning, yet the overwhelming majority of those episodes come from stationary, table-mounted arms. That is not a criticism of the corpus; it reflects where the field found it cheapest to collect. But it means a policy pretrained on that data has seen a great deal of reaching and grasping and very little of walking up to a counter it has never stood at and deciding where to stop.
What changes when the base can move
Bolt the arm to a wheeled or legged base and three things expand at once.
The state gets bigger. A fixed-base arm's state is essentially its joint angles, the gripper, and the object. A mobile system adds the base pose in the world, its heading and velocity, and the relationship between the base and the arm. More degrees of freedom means more of the space to cover, and coverage is exactly what demonstration data buys.
The coordinate frames multiply and drift. On a bench, an object's position can be written once in a world frame and trusted for the whole session. On a moving base there is a world frame, a base frame, an arm-base frame, a head-camera frame, and a gripper frame, and all of them move relative to each other. Worse, the base's own estimate of where it is drifts. Wheel odometry slips, a legged gait introduces sway, and SLAM localization can be off by centimeters. A grasp target written in world coordinates inherits that error. Data captured in a body-relative frame, where the target is expressed relative to the head or the hand, sidesteps the drift entirely, because the relationships a policy actually uses do not depend on global position.
The viewpoint moves with the body. A wall camera can watch a fixed arm forever. It cannot follow a robot that roams a building. The only camera guaranteed to be present at manipulation time is the one on the robot itself, and that camera arrives at the object by whatever path the base took. What is visible at the grasp depends on the approach, so the approach has to be in the data too.
| Dimension | Fixed-base manipulation | Mobile manipulation |
|---|---|---|
| State to capture | Arm joints, gripper, object | Arm, gripper, plus base pose, heading, and velocity |
| Coordinate frames | World and robot frame coincide, stable | Multiple moving frames, base localization drifts |
| Viewpoint | Can use fixed external cameras | Onboard only, moves with the body, set by the approach |
| Episode length | Seconds to about a minute | Often minutes, navigation plus manipulation |
| Capture difficulty | Contained, repeatable | Larger, longer, harder to synchronize and reset |
The handoff is where it breaks
Classical robotics keeps navigation and manipulation in separate boxes: a planner drives the base to a target pose, then a manipulation controller takes over. The seam between them is where mobile manipulation quietly fails. Stop half a meter too far and the shelf is out of reach. Approach at the wrong angle and the grasp the policy learned no longer lines up. The base pose at the end of navigation is the initial condition for everything the arm does next, and a demonstration that treats the two phases as unrelated clips throws that dependency away.
Good mobile demonstrations record the coupling instead. They capture the base pre-positioning itself for the reach, the arm beginning to extend before the base has fully stopped, the whole-body coordination that lets a robot lean its torso to add a few centimeters of reach rather than take another step. This is why whole-body teleoperation rigs matter: systems in the mould of Mobile ALOHA, developed at Stanford, let one operator drive the base and both arms together, so the recorded trajectory contains the coordination and not a stitched-together sequence of navigate-then-grasp. Research from groups like the Stanford IRIS lab and a steady stream of cs.RO work has pushed this whole-body framing precisely because the naive module boundary loses the most important information.
Where the robot stops is the first decision of the grasp, not the last decision of the walk. Data that separates the two teaches a policy to approach and manipulate as strangers.
Why a body-worn camera fits
Humans are the reference mobile manipulators. We cross a room, open a drawer, and take out a spoon without a thought, coupling locomotion and manipulation so tightly that we never notice the handoff. Capture that from the person's own point of view, a head camera and a camera on the wrist, and the recording has a property that suits a mobile robot exactly: the viewpoint moves with the body, the same way the robot's onboard cameras will.
This is the deeper reason egocentric capture and mobile manipulation belong together. A body-worn camera walks up to the counter, so it records the approach, the changing scale of the target, the moment the hand enters the frame from below. It is naturally body-relative, so it does not depend on a perfect global position that a real robot will never have. And because a person navigating to a task and then performing it is one unbroken stream, the coupling between getting there and doing the task is captured for free, in the one place it is hardest to fake after the fact.
Deployed systems make the point concrete. Agility Robotics built Digit, a bipedal robot that walks to totes and moves them in warehouses; Google DeepMind has run manipulation policies on mobile platforms in kitchens and offices, where the robot has to drive itself within reach before it can act. In both cases the robot's sensors are on the robot, moving with it, seeing what a body-worn camera sees, and not what a tripod across the room would.
The base is not a footnote
It is tempting to treat the drivetrain as plumbing under a manipulation problem that was already solved on a bench. The data says otherwise. When the base moves, the state grows, the frames drift, the viewpoint travels, and the episode stretches to include the walk that set up the grasp. A robot that has only ever learned from an arm bolted to a table will approach the world as if the world came to it. The ones that walk to the object, and grab it once they arrive, will have learned from data recorded the same way they move.