How to actually measure a robot policy
A robot policy can post 90% on a benchmark and still fail in the field. The pitfalls of policy evaluation, and how to test one honestly.
A robot policy posts 90% on a benchmark and the headline writes itself. Ninety percent of what, though: measured how, over how many trials, on which objects. A number on a leaderboard is a compression of an experiment, and most of what you would need in order to trust it was thrown away in the compression.
Evaluation is the quiet crisis of robot learning. Almost everyone agrees the policies are getting better. Almost no one agrees on how to prove it. A manipulation policy is not a language model you can grade overnight on a frozen test set. Every trial costs a physical robot, a reset, and a human to run it again, and the world refuses to hold still between runs: the light shifts, the object rolls a centimeter, the gripper wears.
So this piece takes a position. Most reported robot success rates are closer to anecdotes than to measurements, and the gap is not carelessness. Honest evaluation is expensive, slow, and unflattering, which is exactly why it gets skipped. The teams whose policies survive contact with the real world are the ones who treat evaluation as a first-class experiment: enough trials to carry a confidence interval, conditions the policy never trained on, and failure modes reported instead of buried.
The number is a compression, and it drops the error bars
Start with the statistics, because they are unforgiving and routinely ignored. Robot evaluations report a binary outcome, success or failure, over a handful of trials. Score a policy at 8 out of 10 and it feels like a solid B. Run the binomial confidence interval and the true success rate could plausibly sit anywhere from about 50% to 95%. Ten trials simply cannot resolve the difference. Worse, over ten trials each you cannot reliably distinguish a policy that scored 8 out of 10 from one that scored 6 out of 10: the intervals overlap almost completely.
This is not a nitpick, it is the difference between a demo and a measurement. Toyota Research Institute has been unusually blunt about it in its Large Behavior Models work, where evaluation is run as a proper experiment: many trials, blind and randomized A/B comparisons so the operator's hope does not leak into how a scene gets reset, and statistical tests before a change is declared an improvement. None of that is glamorous. All of it is what separates a policy that got lucky on Tuesday from one that is genuinely better.
Simulation benchmarks reward the wrong thing
Simulators are cheap, fast, and perfectly reproducible, which is exactly why they are seductive and exactly why they mislead. A policy can climb a simulation benchmark by fitting the quirks of one physics engine and one renderer rather than the task. The contact model is an approximation; overfit it and you have a policy tuned to a world that does not exist.
The gap is not uniform. Simulation transfers well for locomotion and whole-body control, where rigid-body dynamics are modeled faithfully and researchers have pushed hard on domain randomization. It transfers poorly for the contact-rich manipulation people most want: soft, deformable, or articulated objects, where a simulator that cannot render a folding cloth or a slipping grasp will happily train a confident policy that fails on the real thing. Much of the sim2real literature on Berkeley BAIR and in the cs.RO preprints circles the same caution: a simulation score is a hypothesis about the real world, not a measurement of it. A benchmark that never touches a real gripper measures how well you overfit a physics engine.
The task set is doing more work than the metric
Before you argue about the metric, look at what it was computed over. The task set silently decides the result. Evaluate on the same objects, the same scenes, and the same lighting the policy trained on and you have measured memorization, not competence. It will look solved right up until the towel is red instead of blue or the sun moves across the table.
Shared benchmarks help, and the field has built good ones. Open X-Embodiment pooled evaluation across many robots and institutions precisely so results could be compared on common ground. But even a shared benchmark can be gamed by reporting only the friendly tasks, and a single aggregate success rate hides which conditions carried the score. The honest protocol is dull and hard to fake: hold out objects, scenes, and lighting the policy has never seen, fix the trial count in advance, and report per-condition rather than one flattering average. If the held-out number is far below the held-in number, that gap is the finding, not a footnote.
Success rate hides how it succeeded, and how it failed
Binary success is a lossy metric even when the statistics and the task set are honest. A policy that squeaks through with a fumbling near-miss grasp and one that acquires the object cleanly on the first try score identically, yet only one of them will survive a millimeter of added clutter. Success rate records the outcome and erases the margin.
Richer evaluation reports more than a bit. Partial credit for task progress separates a policy that got halfway from one that never started. A robustness margin, meaning how much perturbation the policy tolerates before it breaks, predicts field behavior far better than clean-room success. Most valuable and most neglected is recovery: does the policy notice a mistake and correct it, or does one slip cascade into a pile of failures. Work from groups like Stanford IRIS on imitation learning has long made the point that robustness and recovery, not flawless demonstrations, are what generalize. Reporting the failure modes is not an admission of weakness, it is the most useful information an evaluation produces.
| Evaluation approach | What it measures | How it misleads |
|---|---|---|
| Simulation benchmark | Behavior in a modeled world, cheaply and reproducibly | Rewards overfitting the physics engine and renderer; contact and deformables break the transfer |
| Small real-world trial set | Actual on-robot success | Confidence interval so wide it cannot separate two policies |
| In-distribution task suite | Competence on trained objects and scenes | Scores memorization; collapses when objects, clutter, or lighting change |
| Binary success rate | Whether the task completed | Ignores margin, robustness, and recovery; a near-miss ties a clean grasp |
| Cherry-picked or best-of-N run | The ceiling on a friendly task | Reports the best trial, not the expected one; a highlight reel, not an average |
A benchmark number tells you how a policy did on the exact thing it was scored on. Whether it will work in the real world is a separate question, and only held-out trials, counted honestly, can answer it.
A good evaluation set is captured data, not a vibe
All of this converges on an inconvenient requirement. Held-out objects, held-out scenes, controlled lighting, and repeatable resets do not appear on demand. To function as a fair test, an evaluation set has to be captured, instrumented, and documented as carefully as the training data, otherwise it cannot be reproduced next quarter and its numbers cannot be compared across teams. An evaluation you cannot rerun is a story, not a benchmark. The uncomfortable truth is that building a hard, honest test set is the same craft as building a good training set, and it is roughly as expensive.
So the next time a robot policy is reported at 90%, ask the three questions the number buried: how many real trials stand behind it, on what held-out conditions, and what did the failures actually look like. If the answer is a single figure over a friendly task set, you are watching a highlight reel. If the answer arrives with an interval, a held-out gap, and an honest account of the failures, you are looking at a measurement, and those are far rarer than the leaderboards suggest.