Observation and Action Spaces: The Quiet Decisions
The action space and observation space you pick decide what a dataset can ever teach. A field guide to the quiet choices that outlast the robot.
Two labs record the same chore, wiping a table. One logs the end-effector pose at 10 Hz. The other logs raw joint angles at 100 Hz plus wrist force. Same skill, same class of robot, and yet their datasets can barely train each other's policies. The gap is not the task. It is the choice of observation and action space, made in an afternoon and then paid for over the entire life of the data.
These two choices are the quietest decisions in a robot-learning project. Nobody writes a blog post the day they pick an action space. And yet that pick decides what can generalize, what can merge with other datasets, and what has to be thrown away later. It deserves far more deliberation than it usually gets.
What space means here
The observation space is the set of everything a policy is allowed to see: which cameras, at what resolution, plus whatever proprioceptive and tactile channels you include. The action space is the set of everything a policy is allowed to command: joint targets, end-effector motions, gripper signals, and how each is encoded. Both are design decisions, not facts about the robot. The same arm can be driven through at least three different action spaces, and each one produces a subtly different dataset from identical motion. A joint-position log, an end-effector-pose log, and a tokenized log of the very same wipe are three different datasets that a downstream team cannot freely interconvert after the fact.
The action space decision
Start with actions, because the action space is the one people underestimate most. A handful of options dominate real datasets.
- Joint position. Command each joint a target angle. Precise, hardware-specific, and nearly impossible to transfer to a robot with different kinematics.
- Joint torque or velocity. Lower-level still, great for dynamic and contact-rich motion, and the least forgiving to capture cleanly.
- End-effector pose. Command where the hand should be, absolute in some frame. More portable across bodies, at the cost of hiding the arm's redundancy.
- End-effector delta. Command a small change in hand pose each step. The current favorite for cross-embodiment data, because a delta means roughly the same thing on many arms.
- Tokenized actions. Discretize any of the above into tokens a language-style model can emit, which is how many Vision-Language-Action models bolt control onto a transformer.
The reason this matters so much is pooling. The Open X-Embodiment effort had to reconcile dozens of datasets that each chose their own action space, and much of the hard work was translating them into a shared representation without corrupting the meaning. DROID made a deliberate bet on a consistent action and observation format across a large teleoperation fleet, precisely so its data would compose.
| Action space | Best for | Weakness | Transfers across bodies? |
|---|---|---|---|
| Joint position | Precise, repeatable motion on one robot | Locked to that robot's kinematics | Poorly |
| Joint torque or velocity | Dynamic, contact-rich control | Hard to capture and imitate cleanly | Poorly |
| End-effector pose | Portable manipulation targets | Hides arm redundancy and self-collision | Moderately |
| End-effector delta | Pooling data across many arms | Drift accumulates without good framing | Well |
| Tokenized actions | Vision-Language-Action models | Discretization error at fine scales | Depends on the tokenizer |
The observation space decision
The observation space carries the same kind of hidden leverage. Three sub-choices do most of the damage, or most of the good.
Which modalities
RGB alone is cheap and often enough for coarse tasks. Add depth and you help geometry. Add proprioception and force and you finally give a policy the information a human uses without thinking during an insertion. Datasets like RH20T were built specifically to carry force and multi-view vision together, because a wipe or a plug-in is barely learnable from a single RGB stream.
Which frame
An image from a wrist camera and an image from a tripod teach different things. A pose expressed in the robot base frame and the same pose in a camera frame are not interchangeable in the data. Mixing frames without recording the transforms is one of the most common ways a dataset quietly becomes unusable.
How much history
A single frame is Markov and easy to shuffle. A short window of frames supports memory and velocity estimation but forces you to keep sequences intact. The observation space is where you commit to that, whether you realize it or not. Get this wrong and you find out late, when a model that needed velocity cannot recover it from frames that were logged too far apart.
You do not choose an action space once. You choose it for every model that will ever train on the data, so the only safe rule is to pick the representation that throws away the least.
Why merging datasets is so hard
All of this converges on the hardest problem in the field's data supply: pooling. When two datasets disagree on action space, observation modalities, coordinate frames, or control rate, merging them is not a file-format chore. It is a semantic translation that can silently distort what the robot was actually doing. The compatibility of a policy foundation like NVIDIA's GR00T with many robot bodies rests on choosing representations that survive this translation. Where the representation is lossy, the merge quietly injects noise that no downstream training run can remove.
This is why control rate belongs in the same conversation. A dataset sampled at 10 Hz and one sampled at 100 Hz do not just differ in size; the slower one has thrown away the fast corrections that a contact-rich skill lives on. Resampling can align the timestamps, but it cannot invent the frames that were never taken.
The task is easy, the space is forever
The observation and action space look like plumbing, and that is exactly why they are dangerous. They are chosen early, by whoever is closest to the hardware, and they lock in what every later model can learn. Choose a narrow action space and you get a clean dataset that ages into a dead end. Choose a rich, well-framed, time-aligned representation and you get data that keeps paying out as policy architectures change under it. The task is the easy part. The space you record it in is the decision that lasts.