Language-Conditioned Manipulation, Grounded in Data

Say "hand me the small screwdriver, the one next to the tape" to a person and they get it right without a second thought. Say it to a robot and every word turns into a problem. Which screwdriver counts as small? Where exactly is "next to"? What does "hand me" mean for a two-finger gripper that cannot flick its wrist the way a hand can?

Language is the cheapest interface we have for commanding a robot. That is the appeal, and the trap. The words arrive with none of the physical grounding needed to act on them. The instruction is easy. Turning it into a specific object, a specific grasp, and a specific path is the hard part.

Here is the claim worth defending. The bottleneck in language-conditioned manipulation is not the language model. It is the paired language-action data that ties words to motion. We have an abundance of the first and a shortage of the second.

The instruction is easy, the grounding is hard

Modern vision-language-action (VLA) models inherit much of their competence from web-scale pretraining. A model that has read the internet already knows a screwdriver is a tool, that tape comes on a roll, that "small" is relative. That prior is real and useful. Google DeepMind's work on Gemini Robotics leans on it, carrying broad visual and linguistic knowledge into a control policy.

But semantics is not grounding. Knowing what a screwdriver is does not tell you where this one sits on this cluttered bench under this light, or how much force it takes to lift it without a slip. That mapping, from a referring phrase to a pose in the robot's own frame, has to be learned from demonstrations that pair the phrase with the motion. Text pretraining does not supply it for free.

So a policy can narrate a task perfectly and still fail to do it. Its words are grounded in other words. They are not grounded in torque, contact, and geometry.

Where the language actually enters the policy

There is no single place an instruction plugs into a control policy, and the design choice decides the data you need. Three patterns dominate.

• Language as a goal token. The instruction is encoded once and handed to the policy as a fixed vector. Simple. It works when one instruction maps to one skill.
• Cross-attention over the words. The policy attends to individual tokens at each step, so "the one next to the tape" can steer perception frame by frame. More expressive, and hungrier for data.
• Language as a plan, then low-level control. A high-level model turns the instruction into sub-goals, and a fast policy executes each. Physical Intelligence has written about splitting a slow deliberate layer from a fast reactive one on the Physical Intelligence blog.

Each pattern feeds on a different slice of data. A goal-token model learns from coarse, one-line episode labels. A cross-attention model wants tight alignment between clauses and moments. The hierarchical split wants both, at two timescales.

Instruction granularity is a data decision

The most consequential choice is granularity: how finely the language is tied to the action. It reads like an annotation detail. It decides what the model can generalize to.

Most large corpora sit at the coarse end, because coarse is cheap. That is a problem. A model trained only on "make a coffee" has no clean route to recombine "open" and "pour" into a task it never saw. Step-level labels are where compositional generalization lives, and they cost real annotation work.

What paired language-action data actually requires

Collecting the demonstration is half the job. Attaching the right words to the right moments is the other half, and most datasets are thin there. Three requirements recur.

• Temporal alignment. "Now twist" has to line up with the frames where the twist happens, not the whole clip. Loose alignment teaches the model to ignore timing.
• Hindsight relabeling. A trajectory recorded for one instruction is a valid demonstration for many. "Reach for the mug" is also, after the fact, "move up and to the left". Relabeling one demo with several true instructions multiplies its worth.
• Referring diversity. The same object should be named many ways: "the red one", "the mug", "the thing on the left". Without that spread, the policy overfits to phrasing.

Public corpora show the gap. Open X-Embodiment aggregates roughly a million real-robot trajectories across many embodiments, and its language annotations are mostly one short instruction per episode. Wide coverage. Thin on the fine-grained, multiply-relabeled alignment a cross-attention policy would prefer. Systems like NVIDIA Isaac GR00T are built to consume paired vision-language-action data, which makes that alignment the binding constraint, not the model.

The scarce resource in language-conditioned manipulation is not language understanding. It is aligned language-action pairs at the right granularity.

How you know the grounding held

There is a cheap test for whether a policy grounded the instruction or memorized it. Change one word. Swap "red" for "blue", "left" for "right", "mug" for "bottle", and watch what the hand does. A grounded policy tracks the change. A memorizing one keeps reaching for the object it saw most in training. Evaluations that vary the phrasing, not just the scene, are the ones that predict real deployment.

This is also why corrective data matters. A dataset of only clean successes never shows the policy what "no, the other one" looks like. The demonstrations that teach grounding include the reach that started wrong and got corrected mid-motion.

Give a robot an instruction and the words are the easy part. Ask it to act and the missing piece is not comprehension. It is a demonstration that once tied those exact words to a real motion, and enough of them, phrased enough ways and aligned tightly enough in time, that the policy learns the mapping instead of the sentence. Build that data and language becomes a control surface. Skip it and language stays a demo.