Sensorimotor Loops in Manipulation

Ask a robot to pick up a mug, then cut its cameras the instant before contact. If the policy still succeeds, it was not really watching. It memorized a motion. A robot that actually manipulates the world runs a tight loop: sense, decide, act, sense again, many times per second. Break that loop and the skill collapses.

This is the sensorimotor loop, and it is the heart of manipulation. Perception drives action, action changes the world, and the changed world feeds back into perception. The loop is not a diagram on a slide. It is a timing contract, measured in milliseconds, and it decides whether captured demonstration data teaches anything useful at all.

Perception, action, and the clock between them

A manipulation policy maps observations to actions. Stated that flatly, it sounds static. It is not. The value of an action depends on how fresh the observation was when the action was chosen. Feed a controller a camera frame that is 200ms stale and it is steering by a photograph of the past.

Every part of the loop runs at its own rate. A wrist camera might stream at 30 fps. A force-torque sensor can report at several hundred Hz. The low-level controller might close its joint loop at 500 Hz or 1 kHz. Human reaction to a visual event is on the order of 200ms, which is why teleoperation systems build in matching delays. These rates are not trivia. They define what the policy can react to, and how quickly.

Why capture either keeps the loop or destroys it

Here is the failure that ruins otherwise good datasets. You record beautiful video and clean joint traces, but you log them on separate clocks and never align them. Now the training pipeline cannot say which observation the human was reacting to when they applied a force. The causal link, the whole point, is gone.

Preserving the loop means three things. Synchronize every stream to a common clock, so a force spike lines up with the frame that caused it. Sample fast enough that brief contact events are not smeared across a single reading. And log the action next to the exact observation that drove it, not the observation from two frames later.

Contact is the unforgiving case. A peg touching a hole, a finger finding an edge: these events last a few milliseconds and carry most of the information. Research groups working on contact-rich skills, from Berkeley BAIR to the teams building large behavior models at Toyota Research Institute, keep returning to the same point. Without high-rate, aligned force and proprioception, the policy is guessing about the moment that matters most.

There is a human factor here too. When a person demonstrates a delicate insertion, they are running their own sensorimotor loop, adjusting grip and angle from what they feel and see. Capture that discards the timing discards the very signal that made the human good at the task. The finger pressure at the instant of contact, aligned to the frame that showed the misalignment, is the lesson. Strip the alignment and you keep the motion but lose the reason for it.

A demonstration is not a video with some numbers attached. It is a causal record of what a human sensed and how they responded, and only aligned capture keeps that causality intact.

Latency is a design parameter, not an accident

Once you accept that the loop has a clock, latency becomes something you design for rather than something you apologize for. If your deployed robot will run inference at 10 Hz onboard, capturing demonstrations that assume instant reaction teaches a rhythm the robot can never reproduce. The training data should reflect the loop the policy will actually live in.

Physical Intelligence has written about the practicalities of running real robot policies at usable rates, and the recurring theme is that timing and closed-loop feedback are first-class concerns, not afterthoughts. You can read more on the Physical Intelligence blog. The NVIDIA GEAR Lab research makes a parallel case from the model side: general physical intelligence depends on models that react to fresh sensory input rather than replay a fixed plan.

How synchronization actually happens

Saying align every stream is easy. Doing it is a hardware and software discipline. The cleanest approach hangs every sensor off a shared time source, so each frame and each force sample carries a timestamp from the same clock rather than from whatever laptop happened to receive it. Where a shared clock is impossible, teams fall back on a synchronization event, a sharp signal visible in several streams at once, and align to it in post.

The subtle failures are the ones that survive a casual look at the data. A camera that timestamps on arrival rather than on exposure introduces a variable delay that depends on system load. A force sensor logged over a congested bus arrives late and jittery. Each error is small. Chained through a control loop that reacts in tens of milliseconds, small is enough to teach a policy the wrong cause for an effect.

This is why capture rate and capture timing are not the same specification. A stream can be fast and still be misaligned, and a slow stream with honest timestamps can be more useful than a fast one without them. The number that matters is not frames per second in isolation. It is how tightly every stream agrees on when each sample happened. None of it shows up in a dataset's headline stats, which is exactly why so many buyers skip the check until a trained policy starts behaving strangely.

Capture the loop, not the snapshot

The temptation is to treat a demonstration as a stack of independent recordings you can merge later. The physics says otherwise. Manipulation is a loop, and the information lives in the timing between sensing and acting. Get that alignment right at capture time and the data is worth training on. Get it wrong and no amount of scale will put the causality back.