Whole-body control: how humanoids coordinate legs and arms
A humanoid must balance on two legs while it manipulates, so every reach shifts its own center of mass. Why whole-body control is hard, and data-starved.
Stand up and reach for something on a high shelf. Notice what your legs do while your arm is still moving. Your ankles tighten, your hips slide back a few centimeters, your weight rolls onto the balls of your feet. You did not decide any of that. Your body solved a physics problem so that the moment your fingers close on the object, you are not already falling toward it.
A robot arm bolted to a workbench never has this problem. Its base is a fixed point in the universe. Push on it and it does not move; reach with it and nothing tips over. That is why most of what we call robot manipulation, and most of the data behind it, quietly assumes a base that cannot move.
A humanoid throws that assumption away. It has to hold itself upright on two feet while it does the very thing that threatens its balance. This coupling, between staying up and getting work done, is the reason whole-body control is one of the harder problems in the field, and the reason the data to learn it is scarce.
The coupling nobody can avoid
The technical name for the difficulty is that a legged robot is underactuated. There is no motor between its feet and the floor. The only way it can push its center of mass around is indirectly, through contact: the friction and normal forces where each foot meets the ground. Every arm motion changes the picture. Swing a heavy tool out to the side and your center of mass drifts toward it. Extend both arms to catch a falling box and the reaction shoves your torso back. On a fixed arm these effects vanish into the bolts in the table. On a humanoid they land on two small contact patches that also have to keep the machine standing.
So a reach is never just a reach. It is a request to move mass, and the legs have to answer it before the hand arrives. Manufacturers have made the balancing half look almost casual. Boston Dynamics has shown Atlas running, jumping, and catching itself after a shove, and Agility's Digit walks warehouse floors and carries totes for hours. The open question is what happens when the same body has to balance and perform fine, forceful manipulation at once, with the two goals actively fighting each other.
On a humanoid a reach is not a reach. It is a request to move mass across the body, and the legs have to grant it before the hand arrives.
Solving it by model
The classical answer is to write the physics down and let an optimizer honor it. Whole-body control treats the robot as one coupled system of dozens of joints, then solves, many times a second, for the joint torques that best track what the arms want while respecting what the legs need. The constraints are hard, literally: the feet must not slip, the contact forces must stay inside the friction cone, the projected center of mass must sit over the support region.
Model predictive control, or MPC, adds time. Instead of solving for this instant alone, it rolls the dynamics forward over a short horizon, a fraction of a second, and picks the action sequence that keeps the robot feasible across all of it, then re-solves at the next tick. This is how a machine can lean into a push before it topples, because it saw the fall coming a few hundred milliseconds ahead. The strength of the approach is that it never has to be taught the concept of balance. Balance is a constraint it obeys by construction. The weakness is that it needs a good model of contact, and contact is exactly where models are worst: the instant a foot slips or a hand jams against a stuck drawer, the tidy equations stop describing reality.
Solving it by learning
The newer answer is to skip the hand-written model and let a policy discover the coordination from experience, usually millions of falls inside a physics simulator. Reinforcement learning in simulation now routinely produces controllers that walk, recover, and climb, and the same recipe is being pushed toward loco-manipulation: policies that move the base and the hands as one action rather than treating walking and grasping as separate modes handed off between two controllers.
Where it gets interesting is the merger with foundation models. NVIDIA's Isaac GR00T program aims at exactly this, a general humanoid policy trained on human video, teleoperation, and simulation, that outputs whole-body action rather than arm-only trajectories. A learned policy has the opposite tradeoff to the optimizer. It handles messy contact gracefully, because it met a million messy contacts in training, but it offers no guarantee: it does what its data taught it, and outside that distribution it can simply fall. Which brings the argument back to the data.
Why the data is the bottleneck
Consider what a demonstration has to contain for each of these approaches. A fixed-arm manipulation clip needs the arm's joint trajectory and a camera; the base is a constant you can ignore. A whole-body demonstration has to capture the legs, the torso, the shifting contacts underfoot, and the forces flowing through the whole kinematic chain, all synchronized, because the leg motion is not context around the task. It is part of the task.
| Dimension | Fixed-arm manipulation | Whole-body humanoid |
|---|---|---|
| State to track | Arm joints plus gripper, roughly 7 numbers | Legs, torso, both arms and hands, on the order of 30 to 50 |
| Base assumption | Fixed and known, safe to ignore | Floating and balancing, must be measured |
| Contacts that matter | Fingertips on the object | Fingertips plus both feet, all changing |
| Balance coupling | None, the table absorbs it | Every arm motion feeds back into stability |
| Capture setup | One arm, bench camera, teleoperation rig | Full-body motion capture plus force, on a moving person |
| Relative data cost | Low, and widely shared | High, and mostly missing |
This is why the large open manipulation corpora, the ones that made cross-embodiment learning possible, are overwhelmingly fixed-base. Pooling arm demonstrations across labs was already a heroic effort. Pooling synchronized whole-body, contact-rich humanoid demonstrations is a harder problem the field is only starting to face. The state space is bigger, the capture rig is heavier, and a person wearing it still has to keep their own balance while performing the task naturally. Reviews across recent robotics literature and reporting in outlets like IEEE Spectrum keep circling the same gap: the hardware to act is arriving faster than the demonstrations to train it.
Whole-body control is not manipulation with extra joints. It is a different problem, because the thing being controlled is also the thing keeping itself upright, and the two cannot be pulled apart. Whether the answer ends up being an optimizer that honors the physics, a policy that learned the physics, or the near-certain blend of both, all of it rests on the same foundation: someone has to record what a whole human body actually does when it walks up to a task and gets it done. That recording is the scarce part, and it does not yet exist at the scale the robots will need.