Continual Learning for Deployed Robots

A warehouse runs forty humanoid robots on the same picking task. On Tuesday a new product arrives, wrapped in a slick film that fools the old grasp. You collect fresh demonstrations, fine-tune the policy overnight, and push the update. On Wednesday the robots handle the new product beautifully, and drop the old one they had handled for months. You just met catastrophic forgetting.

Deployed robots do not get to freeze their skills. The world keeps changing: new objects, new stations, worn grippers, seasonal stock. A fleet that cannot keep learning goes stale. A fleet that learns carelessly forgets. Continual learning is the discipline of doing the first without the second, and it lives or dies on the data pipeline behind it.

Why updating a policy is dangerous

Neural policies store skills in shared weights. Train hard on a new task and gradient descent will happily overwrite the parameters that encoded an old one. The model has no built-in sense that the old skill still matters. This is not a quirk of one architecture. It is a general property of how these networks learn, and it gets worse as a single model carries more skills.

The naive fix, retrain from scratch on everything each time, works but is slow and expensive, and it assumes you still hold all the old data in a usable form. The interesting engineering is in the middle: how to fold new experience into a deployed policy while proving the old behavior survived.

Replay is the workhorse. You keep a curated buffer of past demonstrations and interleave them with new ones, so the gradients that teach the new skill are balanced by gradients that defend the old. Replay only works if you actually kept the old data, versioned and queryable, which is a data pipeline problem long before it is a training problem.

A fleet that learns from its own deployment is a flywheel. A flywheel with no brakes is just a faster way to drift off course.

The pipeline behind a fleet that learns

Continual learning at fleet scale is a loop with more moving parts than the training step everyone pictures. It runs roughly like this.

• Collect: robots log their own runs, successes and failures, from the deployment floor.
• Curate: filter, dedup, and label the incoming stream so you are not training on noise or near-duplicates.
• Retain: version the corpus so every past skill still has representative data to replay.
• Evaluate: gate every candidate update against a fixed suite that includes the old skills, not just the new one.
• Redeploy: ship only if the gate proves no regression, and keep the lineage to roll back.

The failure most teams underestimate is the feedback loop. A deployed policy shapes the data it collects next. If the robot avoids a hard grasp, it stops generating examples of that grasp, and the next model is weaker at it still. Companies shipping real humanoids, from Figure to 1X and Agility, are effectively running these loops on real floors, and the hard part they share is not the neural net. It is keeping the data honest as the fleet changes what it sees.

The flywheel can spin the wrong way

The optimistic story about deployed robots is a data flywheel: more robots produce more data, better data makes better policies, better policies get deployed more widely, and the loop compounds. It is a real effect. It is also only half the picture, because the same loop can compound a mistake.

A deployed policy is not a neutral observer of the world. It decides which situations it enters, and therefore which data it generates. Send a fleet to pick reachable items and it learns reachable items ever better, while the awkward reach in the corner never enters the training set at all. The dataset drifts toward what the current policy already does well, and the blind spots harden. This is distribution drift driven from the inside, and volume does not fix it. It can make it worse.

The counter is deliberate collection. You have to spend effort capturing the cases the fleet avoids, sometimes by teleoperating through them, sometimes by sourcing demonstrations of a human doing the hard version. Left alone, a self-collecting fleet optimizes for its own comfort, not for coverage. Reporting on the sector keeps circling the same lesson: the gap between a scripted demo and a dependable daily shift is rarely the model architecture. It is whether the team controls what its fleet learns from, or lets the fleet quietly narrow its own diet.

Evaluation is the brake

If replay is the engine, evaluation is the brake, and continual learning without a brake is reckless. Every update has to clear a held-out suite that exercises the full skill set, so a gain on the new SKU cannot hide a silent loss on ten old ones. That suite has to be versioned too, or it drifts along with the model and stops catching regressions. Reporting on real deployments, from outlets like The Robot Report, keeps surfacing the same reality: the demo is easy, the reliable second month is hard, and the difference is usually the evaluation and data discipline behind the update.

Governance rides along with all of this. Each deployed model is a version, trained on a specific slice of a growing corpus, and a regulated operator has to be able to say which data produced the policy currently on the floor. That ties continual learning back to provenance. If you cannot name the trajectories behind a given update, you cannot audit it, and you cannot cleanly roll it back when the second month goes wrong.

Learning without forgetting is a data problem

It is tempting to picture continual learning as a clever training trick. Most of the real work is elsewhere. It is in retaining the right history, curating the incoming stream, and gating each update against everything the fleet already knows. The robots that keep getting better on a real floor will be the ones standing on a pipeline built for it, not the ones with the cleverest single model.