Reward Models for Robot Learning, and Why Rewards Fail

Here is a task that sounds trivial: reward a robot for putting a dish in a rack. In a simulator you write one line, reward equals one when the dish is seated and zero otherwise, and reinforcement learning does the rest. On a real robot that line is impossible to write, because nothing in the system can reliably tell you the dish is seated.

That gap is the whole problem. Reinforcement learning needs a reward at every step, and for manipulation the reward is exactly the thing you cannot measure. You can measure joint angles. You cannot measure whether the shirt is folded neatly, or whether the plug is fully inserted, without in effect already having solved perception.

So the field is quietly moving the hard part. Instead of hand-writing rewards, teams are learning them, from human preferences, from demonstrations, from video. This changes what data a robotics team actually needs, and it helps to see why hand-designed rewards fail before looking at what replaces them.

Why hand-designed rewards break

Reward engineering works beautifully in games. The score went up, so the reward goes up. Physical manipulation has three properties that break this clean loop.

The reward is unobservable. Success is a statement about the world, such as the cup being upright, full, and on the coaster, and reading that off raw sensors is itself a hard perception problem. If you had a perfect success detector, you would be halfway to a policy already.

Dense shaping backfires. To avoid sparse rewards, engineers add shaping terms: get closer to the cup, small reward; align the gripper, small reward. Robots then exploit the letter of the reward. A policy will happily hover next to the cup collecting proximity reward forever, a behavior known as reward hacking, well documented across reinforcement learning research from groups like Google DeepMind blog.

Contact makes it brittle. Small changes in force or timing flip success into failure, so a reward tuned for one object, one pose, and one day rarely transfers. Every new task becomes a new reward-tuning project. That does not scale to the thousands of skills a humanoid needs.

The reward function is where most of the human effort in robot reinforcement learning secretly lives, and hand-writing it does not scale past a demo.

What a learned reward model does

A learned reward model replaces the hand-written rule with a trained function that scores how good a state or a trajectory is. Feed it an observation, it returns a number. The policy optimizes against that learned score instead of a human-authored formula. Where does the score come from? Three main sources.

Human preferences

Show a person two short clips of the robot attempting a task and ask which is better. Collect thousands of these comparisons, then fit a reward model that agrees with the human ranking. This is the same preference-learning recipe that tuned large language models, applied to physical behavior. It sidesteps the unobservability problem, because a human judges success directly.

The cost is real. Every comparison is a human decision, and raters disagree on messy, mid-attempt clips, so the labels are noisy. But the noise averages out across thousands of judgments, and the model ends up scoring behavior the way people actually do, not the way an engineer guessed they would.

Demonstrations as implicit reward

If you have expert demonstrations, you can infer the reward that makes those demonstrations look optimal, the idea behind inverse reinforcement learning. Here the data itself, the human showing the task, carries the reward signal. Academic groups such as the Berkeley BAIR blog have pushed this line for years.

Video and foundation models as judges

A large vision-language model can be prompted to score whether a frame shows task success, turning a general model into a rough reward function. It is noisy, but it scales, and it needs no per-task engineering. Groups working on generalist physical agents, including the NVIDIA GEAR Lab research and the Physical Intelligence blog, lean on learned and model-based signals rather than hand-tuned rewards.

What this means for the data you collect

The shift from written to learned rewards moves the burden onto data, and it changes the kind of data that has value. If your reward comes from preferences, you need paired comparisons and, crucially, failures to compare against successes. If it comes from demonstrations, demo quality becomes the reward's quality. Either way, a dataset of only clean successes is worth less than a dataset that also captures near-misses, corrections, and clear failures, because a reward model learns the boundary between good and bad from both sides.

There is a subtler point about balance. A reward model trained only on flawless runs never learns what a small mistake costs, so it cannot guide a policy back from the edge. You want a spread: confident successes, marginal successes, and honest failures, each labeled. Curating that spread is a data-collection decision made long before any training run starts.

This is a recurring theme in current robotics work indexed on arXiv Robotics (cs.RO). Reward, policy, and data are not separable problems. Decide how you will score behavior and you have half-decided what to capture.

The reward is now a data problem

Rewards fail in the real world for a reason that is easy to state and hard to fix: the thing you want to reward is the thing you cannot see. Learned reward models do not remove that difficulty. They relocate it, out of an engineer's formula and into a dataset of human judgments and demonstrations. That is progress, because data scales and hand-tuning does not. It also means the reward function is no longer a footnote in a training run. It is a data problem, and whoever captures the right comparisons and failures owns a piece of it.