Multi-Task Learning for Humanoids

Train one humanoid to fold towels. It works. Now teach the same network to load a dishwasher, and watch the towel-folding quietly get worse. Nothing about the folding data changed. The model did, because it now serves two masters at once.

This is task interference, and it is the central tax on multi-task learning. A single policy that covers many skills is what every humanoid team actually wants to ship: one model, many jobs, one thing to maintain and update. But the moment skills share weights, they start competing for those weights. The fight is decided less by the architecture than by the data you feed it.

The comforting story is that skills help each other. Reaching, grasping, and placing share so much structure that learning one should bootstrap the rest. Sometimes that is exactly what happens. Just as often the gradients disagree, and the model splits the difference into a policy that is mediocre at everything.

Positive transfer, negative transfer, and the line between

Multi-task learning rests on a bet: a representation useful for one task is useful for many. When the bet pays off, we call it positive transfer. A model trained on twenty manipulation tasks can pick up the twenty-first from a handful of demonstrations, because the shared backbone already encodes contact, geometry, and object permanence.

When the bet loses, we get negative transfer. Two tasks demand incompatible behavior from the same parameters, and improving one degrades the other. Pouring wants a slow, compliant wrist. Driving a bolt wants a stiff, precise one. Force both through one shared layer and neither gets what it needs.

The deciding mechanism is usually gradient conflict. During training, each task pushes the shared weights in some direction. When two tasks pull in roughly opposite directions, the update cancels, and both stall. Teams building generalist policies, such as the work described on the Physical Intelligence blog and around NVIDIA Isaac GR00T, spend much of their effort keeping that conflict manageable across dozens of skills and several robot bodies.

The data mix is the real control knob

You can attack interference from the model side, with bigger networks, task-specific output heads, or routing. Those help. But the cheapest and most powerful lever is the data mix: how much of each task the model sees per batch.

Sampling is never neutral. If 80% of your corpus is pick-and-place and 2% is insertion, the model will be fluent at picking and clumsy at inserting, no matter how elegant the architecture. The mix encodes your priorities whether you set it on purpose or let raw volume decide for you.

Temperature scaling is the workhorse here. Raise each task's example count to a power between 0 and 1, then sample in proportion to the result. A power of 1 is plain proportional sampling. A power near 0 approaches uniform. The useful settings sit in between, and the best one is empirical, not analytic. You find it by measuring per-task success, not by reasoning about it on a whiteboard.

Task interference is a data-composition problem before it is a modeling problem. You cannot architecture your way out of a corpus that is ninety percent one skill.

Capacity buys peace, but not for free

Scale changes the calculus. A small network forces every task to share every parameter, so conflict is unavoidable. A larger network can hand different subspaces to different skills, and the interference softens. This is one quiet reason the field keeps growing model size even when a single task would never need it.

Mixture-of-experts and task-conditioned routing push the idea further. Send each input to the parameters that suit it, and unrelated skills stop colliding in the same weights. The tradeoff is data. Sparse, specialized capacity is hungry, and it only pays off if each expert sees enough examples to become good at its slice of the problem.

The Toyota Research Institute's work on large behavior models makes the point concretely. A single network absorbing hundreds of skills needs those skills represented densely and cleanly, or the long tail dissolves into noise the model learns to ignore.

Cross-embodiment is another axis of the same problem

Humanoids rarely learn from one robot. Corpora like Open X-Embodiment blend demonstrations across many bodies, and each embodiment is, in effect, one more task to balance. A clip from a parallel gripper and a clip from a five-finger hand describe different action spaces. Mixed carelessly, they interfere. Mixed well, they transfer, and a skill learned on one body carries over to another.

What a good mix demands from the data

Here is the catch the modeling literature underplays. Every strategy in the table assumes you can identify which task a demonstration belongs to, how hard it is, and where it came from. That metadata is not free. It is captured up front, or reconstructed later at painful cost.

To re-balance a corpus you need clean task labels, per-demonstration difficulty signals, and enough provenance to slice by embodiment, environment, and operator. Robotics work from Google DeepMind, described on the Google DeepMind blog, leans on exactly this kind of structured, well-labeled data to train models that span many behaviors without one drowning the rest.

One model for many skills is the right goal. It is also a standing negotiation between tasks that would each prefer the whole network to themselves. Architecture referees that negotiation. The data mix decides who wins. Teams that treat the mix as a first-class artifact, tracked and labeled and adjustable, get generalists that are genuinely good at the tail. Teams that pour everything into one bucket and hope will keep shipping models that fold towels a little worse every time they learn something new.