我们究竟在机器人操作中基准测试什么？
What Are We Actually Benchmarking in Robot Manipulation?
AI总结：
本文通过识别基准测试的四种失效模式（捷径可解性、缺乏统计显著性、渐进过拟合和数据源依赖性），并提出相应诊断方法，对LIBERO、CALVIN、SimplerEnv、RoboCasa和RoboTwin 2.0进行审计，发现多数基准测试存在缺陷，并发布了诊断工具。
摘要：
机器人基准测试分数衡量的是在固定评估设置下的成功率，但通常被当作通用操作能力的证据。我们识别出四种失效模式，每种模式都会削弱或否定基准测试作为该能力有效代理的作用：捷径可解性、缺乏统计显著性、渐进过拟合和数据源依赖性。我们为每种失效模式提出一种诊断方法。我们使用这些诊断方法审计了LIBERO、CALVIN、SimplerEnv、RoboCasa和RoboTwin 2.0。LIBERO和CALVIN未通过多项诊断。RoboCasa和RoboTwin 2.0未通过较少，尽管它们在近期进展声明中出现的频率远低于前者。在LIBERO上，一个没有语言编码器的0.09B探针得分达到或接近报告的最优结果，且大多数报告的性能提升无法证明具有统计显著性。在CALVIN上，在训练范围内随机化块的位置会降低所有测试策略的性能。我们发布了四种诊断方法及其参考实现，供作者和审稿人在将基准测试分数视为进展证据之前使用。代码和工件可在https://ripl.github.io/manipulation_benchmark_audit/获取。
英文摘要：
A robotics benchmark score measures success under one fixed evaluation setup, yet is routinely treated as evidence of general manipulation capability. We identify four failure modes, each of which weakens or invalidates a benchmark's role as a valid proxy for that capability: shortcut solvability, lack of statistical significance, creeping overfitting, and data-source dependence. We propose one diagnostic per failure mode. We audit LIBERO, CALVIN, SimplerEnv, RoboCasa, and RoboTwin 2.0 under these diagnostics. LIBERO and CALVIN fail multiple diagnostics. RoboCasa and RoboTwin 2.0 fail fewer, despite appearing far less often in recent progress claims. On LIBERO, a 0.09B probe with no language encoder scores at or near reported SOTA, and most reported gains are not provably statistically significant. On CALVIN, randomizing block poses within the training range drops performance for every tested policy. We release the four diagnostics with reference implementations for authors and reviewers to apply before treating a benchmark score as evidence of progress. Code and artifacts are available at https://ripl.github.io/manipulation_benchmark_audit/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
