THE COLOSSEUM：用于评估机器人操作泛化能力的基准
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
AI总结：
提出THE COLOSSEUM仿真基准，含20个操作任务和14种环境扰动，评估5个模型，发现成功率下降30-75%，并验证仿真与真实世界结果相关。
摘要：
为了实现有效的大规模真实世界机器人应用，我们必须评估机器人策略如何适应环境条件的变化。不幸的是，大多数研究在环境与训练设置高度相似甚至完全相同的情况下评估机器人性能。我们提出了THE COLOSSEUM，一个新颖的仿真基准，包含20个多样化的操作任务，能够系统地评估模型在14个环境扰动轴上的表现。这些扰动包括物体、桌面和背景的颜色、纹理和尺寸的变化；我们还改变光照、干扰物、物理属性扰动和相机位姿。使用THE COLOSSEUM，我们比较了5个最先进的操作模型，发现它们的成功率在这些扰动因素下下降了30-50%。当多种扰动同时施加时，成功率下降≥75%。我们确定改变干扰物数量、目标物体颜色或光照条件是对模型性能影响最大的扰动。为了验证我们结果的生态效度，我们展示了仿真结果与真实世界实验中的类似扰动相关（R̄² = 0.614）。我们开源了代码供他人使用THE COLOSSEUM，并发布了用于复制真实世界扰动的3D打印物体代码。最终，我们希望THE COLOSSEUM能作为一个基准，用于识别系统性地提高操作泛化能力的建模决策。更多详情请参见https://robot-colosseum.github.io/。
英文摘要：
To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, physical properties perturbations and camera pose. Using THE COLOSSEUM, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades $\geq$75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated ($\bar{R}^2 = 0.614$) to similar perturbations in real-world experiments. We open source code for others to use THE COLOSSEUM, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation. See https://robot-colosseum.github.io/ for more details.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
