Robot Policy Evaluation

• Learning
• Testing
• Failing
• Recovering
• Evaluating

Summary of AI-Generated Content

• RoboLab addresses shortcomings in robotics policy evaluation by enabling robot-agnostic benchmarking with rapid task and scene generation
• The platform integrates advanced diagnostic tools for fine-grained analysis on policy performance and robustness
• RoboLab's competency-tagged tasks ensure comprehensive coverage of manipulation skills and adaptability

Important information may be incomplete in AI-generated content, verify key details

Robotics foundation models have made significant progress, but evaluating them rigorously has become a major challenge

Current benchmarks have limitations, including high setup overhead and issues with visual and task domain overlap

Real-world testing is expensive and difficult to reproduce, making simulation a necessary proxy for large-scale robot evaluations

Visual domain overlap in training and evaluation can lead to models memorizing the setup rather than generalizing

Real2sim approaches can reconstruct photorealistic environments, but per-scene setup can be time-consuming

Benchmark saturation occurs when models quickly max out scores on static task sets, making it hard to distinguish between models

A diagnostic gap exists in current benchmarks, making it difficult to understand why a robot failed or succeeded

Binary success/failure scores do not provide enough information about a robot's performance or failure

Statistical trustworthiness is crucial in evaluating robot policies, as small sample sizes can lead to inaccurate conclusions

The Clopper-Pearson method can be used to construct a binomial confidence interval around a success rate

Most published benchmarks do not run a sufficient number of rollouts to achieve statistical significance

Introducing RoboLab, a simulation benchmarking platform that addresses the issues with current benchmarks

RoboLab is built around three principles: robot benchmarking, bring-your-own-robot, and capability-specific tasks

RoboLab mirrors a real-world setup procedure and allows users to generate novel tasks directly in their workflow

RoboLab tasks are robot- and policy-agnostic, allowing users to make their own design choices

A useful benchmark needs to isolate distinct capabilities, not just measure whether a robot completes a task

Building a generalist robot policy requires solving a long tail of specific tasks, and no single team has abundant data across every embodiment. A lab might have thousands of hours on a Franka arm but almost none on a humanoid, or vice versa. A benchmark tied to one specific robot forces every user into that same data gap, regardless of what they’re actually trying to build or test.

By designing tasks that target specific capabilities, RoboLab ensures broad coverage across the full space of skills a general-purpose policy needs

Evaluating robot policies requires more than just success rates, including metrics that diagnose performance

A useful benchmark needs to isolate distinct capabilities, not just measure whether a robot completes a task. We have observed that general-purpose manipulation draws on at least three separate competencies:

• Visual competency tests whether a policy can recognize and act on perceptual attributes like color, size, and semantic category, such as distinguishing the small red cup from other objects on the table.
• Procedural competency evaluates action-oriented reasoning: stacking objects, reorienting them, or inferring how to interact with a tool.
• Relational competency probes spatial and linguistic logic, including conjunctions (“pick the orange and the lime”), counting, and relative positions like left of or inside.

RoboLab includes a built-in dashboard that surfaces events as they happen during an episode

Real-world deployment rarely offers the clean, controlled conditions of a benchmark

To understand whether a policy is truly robust, performance must be analyzed against increasing complexity in language, scene, and task horizon

Language complexity testing reveals how much a policy depends on exact phrasing versus genuine task understanding

• Graded task scores: Partial credit for completing subtasks within a multi-step instruction, so a robot that grasps the right object but misses the drop target isn’t scored the same as one that does nothing at all.
• Trajectory quality: Measuring motion efficiency via path length and SPARC (Spectral Arc-Length) , a human-aligned metric that captures smoothness through the Fourier spectrum of velocity. Shorter, smoother motions are preferred.
• Speed of execution: Measures end effector velocity, another human-aligned metric that captures the human’s perception that faster motion is preferred.

Task complexity measurement reveals how well a policy sustains accuracy over extended reasoning chains

Sensitivity analysis identifies which environmental variables are most associated with success or failure

Neural Posterior Estimation can be used to estimate the posterior distribution of environmental variables

Robotics benchmarking still lags behind the rest of AI research, and a field-standard benchmarking platform is needed

RoboLab establishes a scalable path toward diagnostic robot evaluation for real-world policies using simulation

For more information about RoboLab, check out the paper and code

RoboLab research powers NVIDIA Isaac Lab-Arena, an open-source simulation framework

Scene complexity

The author thanks several researchers for insightful discussions throughout the work on large-scale robot evaluations

Several papers are referenced for further reading on robotics and AI research

Comments are closed

How sensitive is your robot policy against variations?

Certain environment variations can cause performance drops, but at scale, testing each variable in isolation quickly becomes intractable. Instead, we run evaluations across many scene variations simultaneously and apply sensitivity analysis, which identifies which environmental variables are most associated with success or failure, turning intuitions like “camera placement might matter” into quantified findings.

Given episode rollouts under variation \(\theta\) with observed outcome \(x\) (for example, task success), the posterior distribution \(p(\theta \mid x) \propto p(x \mid \theta)p(\theta)\) characterizes which conditions \(\theta\) are most associated with the outcome \(x\). We estimate this posterior using Neural Posterior Estimation (NPE), which lets us pinpoint exactly which environmental variable is responsible for a given performance drop, rather than guessing at each factor’s impact one at a time.

Robotics benchmarking still lags far behind the rest of AI research, and without a field-standard benchmarking platform, it is difficult to measure progress. As policies grow more capable, success rates alone won’t tell us whether a model truly generalizes or just memorized its test conditions, and that gap will only widen as models improve. The path forward requires evaluation that evolves as fast as the models it measures: benchmarks that expand rather than saturate, metrics that diagnose rather than simply score, and analysis that tells researchers not just how well a policy performs, but how to improve it. RoboLab establishes a scalable path toward diagnostic robot evaluation for real-world policies using simulation.

For more information about RoboLab , check out the paper and code on GitHub. RoboLab was developed by NVIDIA Research, including researchers with affiliations at the University of Sydney and University of Toronto.

RoboLab research powers NVIDIA Isaac Lab-Arena , an open source simulation framework for large-scale policy setup and evaluation. Key RoboLab features are planned for productization in August 2026.

• Website: https://research.nvidia.com/labs/srl/projects/robolab
• Paper: https://arxiv.org/abs/2604.09860
• Code: https://github.com/NVLabs/RoboLab

Acknowledgements

The author thanks Alex Zook, Alperen Degirmenci, Ankit Goyal, Elie Aljalbout, Fabio Ramos, Hugo Hadfield, Jonathan Tremblay, Karl Pertsch, Moritz Reuss, Rishit Dagli, Stan Birchfield (alphabetically listed) for insightful discussions throughout our work on large-scale robot evaluations.

@misc{yang2026benchmarking, title = {How to evaluate real-world policies for general-purpose robots}, author = {Yang, Xuning}, year = {2026}, month = {July}, organization = {Seattle Robotics Lab (SRL), NVIDIA}, howpublished = {https://developer.nvidia.com/blog/how-to-evaluate-general-purpose-robot-policies-for-real-world-deployment{}}, note = {Blog post}, }

Yu, T., et al. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. CoRL 2019. https://arxiv.org/abs/1910.10897

Liu, B., et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023. https://arxiv.org/abs/2306.03310

Zhu, Y., et al. robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. arXiv 2020. https://arxiv.org/abs/2009.12293

Mu, Y., et al. RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins. CVPR 2025. https://arxiv.org/abs/2504.13059

Jain, A., et al. PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies. arXiv 2025. https://arxiv.org/abs/2512.16881

Jangir, Y., et al. RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation. arXiv 2025. https://arxiv.org/abs/2510.23571

Li, X., et al. Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024. https://arxiv.org/abs/2405.05941

TRI LBM Team et al., “A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation”, Science Robotics, 2026, https://arxiv.org/abs/2507.05331

Frazier, D. T., et al. “The Statistical Accuracy of Neural Posterior and Likelihood Estimation.” 2024, https://arxiv.org/abs/2411.12068

Black, K., Brown, N., Driess, D., et al. “\(\pi\) 0: A Vision-Language-Action Flow Model for General Robot Control.” 2024, https://arxiv.org/abs/2410.24164

Pertsch, K., Stachowicz, K., Ichter, B., et al. “FAST: Efficient Action Tokenization for Vision-Language-Action Models.” 2025, https://arxiv.org/abs/2501.09747

Black, K., Brown, N., Driess, D., Esmail, A., et al. “\(\pi\) 0.5: a Vision-Language-Action Model with Open-World Generalization.” CoRL 2025, https://arxiv.org/abs/2504.16054

Beyer, L., Steiner, A., Pinto, A. S., et al. “PaliGemma: A versatile 3B VLM for transfer.” 2024, https://arxiv.org/abs/2407.07726

About the Authors

Comments are closed.