What Is a Robot Policy? Observation to Action
A robot policy maps observation to action. See how output rate, action format, memory, and determinism decide exactly what data you must capture.
Strip the marketing off a humanoid robot and one object sits at the dead center: a function. It takes what the robot senses right now and returns what the motors should do next. Roboticists call that function the policy, and nearly every consequential decision in a robot-learning stack is secretly a decision about the policy's shape.
You can ignore the policy and still ship a demo. You cannot ignore it and still understand why one dataset trains a robust skill while another, larger and more expensive, trains a brittle one. The policy is where data meets behavior, so it is the right place to start.
This piece defines the policy as plainly as it can, then shows how its shape reaches back up the pipeline and decides what you have to capture.
Observation in, action out
At its core a policy is a mapping. Feed it an observation, the robot's current view of the world, and it returns an action, the command that moves the body. Run that loop tens or hundreds of times a second and you get behavior. Everything else, the neural network, the training objective, the dataset, exists only to produce a good version of that one mapping.
Two details make the definition useful rather than trivial. First, what counts as the observation is a choice: pixels from one camera, or pixels plus joint angles plus contact forces. Second, what counts as the action is also a choice: a target joint position, a change in end-effector pose, or a discrete token from a learned vocabulary. Toolkits like LeRobot make these choices explicit in their data schema, which is a good sign the field now takes them seriously.
It helps to say what a policy is not. It is not a plan laid out in advance, and it is not a fixed script. A good policy is closed-loop: it re-reads the world at every tick and can react to a cup that slipped or a peg that jammed. That reactivity is the whole reason imitation from static video is hard, and the reason the observation you feed the policy has to arrive fast enough to matter.
A policy has a shape, and the shape has consequences
Not all policies are built the same, and the differences are not cosmetic. Four axes matter most.
Output rate
A policy that outputs at 10 Hz and one that outputs at 200 Hz are different animals. The slow one can lean on a heavy vision backbone and plan deliberately. The fast one has to be light enough to close a control loop in single-digit milliseconds. Contact-rich skills usually need the fast loop, and that requirement travels all the way back to how fast you have to capture.
Action representation
Absolute joint positions, end-effector deltas, and tokenized actions each imply a different dataset and a different generalization story. An end-effector delta transfers across robot bodies more gracefully than a raw joint command, which matters the instant you try to pool data across embodiments, as Open X-Embodiment found.
Memory
A Markov policy sees only the current frame. A policy with history can remember that it already picked up the screw and is now carrying it. Memory buys you long-horizon behavior and costs you clean, temporally coherent sequences in the data instead of shuffled single frames.
Determinism
A policy that regresses to one action will average two equally good options into a bad one. A generative policy, a diffusion model or a token sampler, keeps the options apart. That choice decides whether your data can hold multiple valid ways to do the same task, or has to be filtered down to one. In manipulation this shows up constantly: there are many good ways to grasp a mug, and a policy forced to pick a single average grasp will often reach for the gap between two of them.
| Design choice | Typical options | What it demands of the data |
|---|---|---|
| Output rate | 10 Hz planning, 200 Hz control | Capture at or above the control rate, tightly time-stamped |
| Action representation | Joint position, end-effector delta, tokens | Consistent frames and calibration across every demo |
| Observation modalities | RGB, plus depth, plus proprioception and force | Every modality synchronized to one clock |
| Memory | Markov, or history and recurrence | Unbroken sequences, not shuffled frames |
| Determinism | Regression, or generative | Permission to keep several valid solutions per task |
Reading a real stack through the policy lens
Look at the current systems this way and they get easier to compare. NVIDIA's GR00T is a foundation policy meant to be adapted to many humanoids, so it leans on end-effector-centric actions and broad pretraining. Physical Intelligence's pi0 and Toyota Research Institute's large behavior models both bet on cloning from big, diverse demonstration sets. The common thread is that each project's headline design decision, the shape of its policy, dictated the shape of the data it had to gather. Much of the recent robot-learning work on arXiv reads as an argument over which policy shape best repays the cost of the data it demands.
Comparing these systems by their benchmark scores tells you little, because they rarely run the same benchmark on the same hardware. Comparing them by policy shape tells you a lot: it predicts which data each one is hungry for, and therefore where each will struggle to scale.
A policy is a bet about what information is enough to choose the next action. The dataset is the evidence for that bet, and no amount of training can add evidence the capture never recorded.
Small definition, large consequences
Ask what a robot policy is and the honest answer is small: a function from observation to action. Ask what follows from its shape and the answer is almost the whole field. Output rate, action representation, memory, and determinism are not implementation trivia. They are the questions that decide, before a single gradient step, what your data has to contain. Get the policy's shape clear in your head first, and the data requirements stop being a mystery and start being a specification. That is the quiet payoff of the abstraction: name the mapping, pin down its shape, and the argument about data stops going in circles.