通过BusyBox评估具身泛化能力
Benchmarking Affordance Generalization with BusyBox
AI总结：
本文提出BusyBox基准，用于评估VLA模型在具身泛化能力上的表现，通过模块化设计挑战模型对新物体的操作能力。
摘要：
视觉-语言-动作（VLA）模型因其泛化能力而受到研究者和实践者的关注。尽管单任务策略仍能提供竞争性性能，VLA能够处理训练集中未见过的命令和环境。虽然视觉和语言空间的泛化对鲁棒的多功能行为至关重要，但VLA需要掌握的关键元技能是具身泛化能力——即能够操控新物体的熟悉物理特性。在本工作中，我们提出了BusyBox，一个用于系统半自动评估VLA具身泛化能力的物理基准。BusyBox包含6个模块，包括开关、滑动杆、导线、按钮、显示屏和调谐器。这些模块可以互换和旋转，以创建具有不同视觉外观但相同具身特性的一系列BusyBox变体。我们实证表明，即使对于强大的开源权重VLA如π_0.5和GR00T-N1.6，跨BusyBox变体的泛化也极具挑战性。为了鼓励研究社区在BusyBox上评估自己的VLA并提出新的具身泛化实验，我们设计BusyBox以便在大多数机器人实验室中轻松构建。我们发布了完整的CAD文件用于3D打印其部件，以及用于（可选）组装其电子元件的物料清单。我们还发布了一个语言标注的演示数据集，该数据集使用常见的双臂Mobile Aloha机器人在标准BusyBox配置上收集。所有发布的材料均在https://microsoft.github.io/BusyBox上提供。
英文摘要：
Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization. Although single-task policies still offer competitive performance, VLAs are increasingly able to handle commands and environments unseen in their training set. While generalization in vision and language space is undoubtedly important for robust versatile behaviors, a key meta-skill VLAs need to possess is affordance generalization -- the ability to manipulate new objects with familiar physical features. In this work, we present BusyBox, a physical benchmark for systematic semi-automatic evaluation of VLAs' affordance generalization. BusyBox consists of 6 modules with switches, sliders, wires, buttons, a display, and a dial. The modules can be swapped and rotated to create a multitude of BusyBox variations with different visual appearances but the same set of affordances. We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights VLAs such as $π_{0.5}$ and GR00T-N1.6. To encourage the research community to evaluate their own VLAs on BusyBox and to propose new affordance generalization experiments, we have designed BusyBox to be easy to build in most robotics labs. We release the full set of CAD files for 3D-printing its parts as well as a bill of materials for (optionally) assembling its electronics. We also publish a dataset of language-annotated demonstrations that we collected using the common bimanual Mobile Aloha robot on the canonical BusyBox configuration. All of the released materials are available at https://microsoft.github.io/BusyBox.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
