Skill Libraries vs End-to-End Policies

There are two ways to build a humanoid that makes breakfast. One trains a single network that maps camera pixels straight to motor commands for the whole task, start to finish. The other keeps a shelf of named skills, crack egg, pour, flip, stack, and a planner that calls them in order. The choice looks like an architecture decision. The bill lands on the data team.

This is one of the oldest live arguments in robot learning: end-to-end policies against skill libraries. It is not settled, and it should not be. The two strategies fail in different places and want different data. Picking one without knowing which data you can actually capture is how projects stall six months in.

The end-to-end bet

An end-to-end policy learns the whole mapping from observation to action as a single function. No hand-designed skill boundaries, no scripted hand-offs. The model works out for itself how to move from lifting the pan to setting it on the burner, and the transition is just more of the same policy running.

The upside is real. Transitions between phases are where hand-built systems break, and an end-to-end model never has to stitch them, because it never split them apart. It can also discover fluid, overlapping motion that a rigid skill sequence would forbid, blending the end of one action into the start of the next. Generalist policies in this spirit are the direction described on the Physical Intelligence blog and behind large multi-task models such as NVIDIA Isaac GR00T.

The cost is data and legibility. A single network covering a long task has to see the whole task, many times, under many conditions, and long-horizon demonstrations are the most expensive kind to collect. When the policy fails, there is no module to blame and no clean place to intervene. You debug a black box by feeding it more data and hoping the failure mode thins out. For a polished demo video that opacity is tolerable. For a system that has to run a full shift, it becomes a liability you carry into every incident review, because a policy you cannot inspect is a policy you cannot easily certify.

The skill-library bet

A skill library takes the opposite stance. Build a set of reusable primitives, each a small closed-loop policy with a clear job, then compose them with a higher-level planner or policy. Breakfast becomes a short program over skills rather than one long reflex.

The advantages are legibility and reuse. A grasp primitive trained once serves hundreds of tasks that involve grasping. When something breaks, you know which skill failed and can retrain it in isolation. Each primitive is short-horizon, so it needs fewer demonstrations to learn well, and data collected for one task strengthens every task that shares its primitives. Teams pursuing broad manipulation, including work discussed by Skild AI, lean on this composability to cover many behaviors without a bespoke model for each one.

The weakness is the seams. Every hand-off between skills is a place to fail. The grasp ends with the object at a slightly wrong angle, and the next skill, trained to assume a clean start, breaks on contact. Managing those transitions, and capturing data at the boundaries where one skill hands off to the next, is the hard part the tidy diagram hides.

The architecture you pick is a bet about where you would rather spend: on model capacity to absorb a whole task, or on data structure to compose it from parts.

What each strategy asks of the data

Here is where the choice stops being abstract. The two strategies want physically different datasets, and you often decide which one you can pursue at capture time, long before a model is trained.

Cross-embodiment corpora sharpen the contrast. A blended dataset like Open X-Embodiment is easier to exploit with modular skills, because a primitive can be trained on whichever bodies performed it, while an end-to-end policy has to reconcile whole-task trajectories across mismatched action spaces. Neither route is impossible. They just ask for different bookkeeping.

The middle is winning

Most serious systems now sit between the poles. Hierarchical policies learn the skills and the orchestration together, so the primitives are not hand-scripted and the transitions are not hand-tuned. Recent manipulation papers on arXiv Robotics (cs.RO) increasingly describe learned low-level skills under a learned high-level controller, keeping the reuse of a library with the fluidity of end-to-end training. It is the pragmatic answer, and it inherits the data appetite of both parents. The honest caveat is that hierarchy does not delete the seams. It moves them inside the model, where they are harder to see but at least no longer hand-tuned by an engineer guessing at thresholds.

Capture for the direction you cannot reverse

The strategic point is about optionality. Continuous, unlabeled, whole-task demonstrations can serve an end-to-end model, but you cannot cleanly split them into primitives after the fact without expensive segmentation and guesswork. Segmented, labeled data with marked skill boundaries can be concatenated back into whole-task sequences whenever you like. One direction is cheap. The other is not.

So the data decision comes before the architecture decision. Capture with clean segment boundaries, per-skill labels, and the transition and recovery moments recorded, and you keep both options open. Capture one long unlabeled stream, and you have quietly chosen end-to-end, whether or not you meant to.

There is no universal winner. End-to-end policies buy fluidity and pay in data and opacity. Skill libraries buy reuse and legibility and pay at the seams. The teams that stay flexible are the ones that stop treating this as a one-time architecture vote and start treating it as a data-capture decision, made early, that keeps both models on the table. Structure the data well, and the architecture argument becomes one you can revisit rather than one you are stuck with.