本文提出面向单臂操作的统一评估生态系统 NEBULA,通过细粒度能力测试与系统性压力测试的双轴协议、标准化 API 和聚合数据集,揭示顶级 VLA 在空间推理与动态适应上的不足。
摘要:
视觉-语言-动作(VLA)智能体的评估受到粗糙的终端任务成功指标的阻碍,这类指标既无法提供精确的技能诊断,也无法衡量其对真实世界扰动的鲁棒性。碎片化的数据格局进一步加剧了这一挑战,妨碍了可复现研究和通用模型的开发。为解决这些局限,我们提出 NEBULA,这是一个面向单臂操作的统一生态系统,可实现诊断式、可复现的评估。NEBULA 采用新颖的双轴评估协议,将用于精确技能诊断的细粒度能力测试与衡量鲁棒性的系统性压力测试相结合。其提供标准化 API 和大规模聚合数据集,以减少碎片化,并支持跨数据集训练与公平比较。借助 NEBULA,我们证明表现最佳的 VLA 在空间推理和动态适应等关键能力上存在困难,而这些能力始终被传统的终端任务成功指标所掩盖。通过同时衡量智能体能做什么以及何时能可靠地完成,NEBULA 为鲁棒、通用的具身智能体提供了实用基础。
英文摘要:
The evaluation of Vision-Language-Action (VLA) agents is hindered by the coarse, end-task success metric that fails to provide precise skill diagnosis or measure robustness to real-world perturbations. This challenge is exacerbated by a fragmented data landscape that impedes reproducible research and the development of generalist models. To address these limitations, we introduce NEBULA, a unified ecosystem for single-arm manipulation that enables diagnostic and reproducible evaluation. NEBULA features a novel dual-axis evaluation protocol that combines fine-grained capability tests for precise skill diagnosis with systematic stress tests that measure robustness. A standardized API and a large-scale, aggregated dataset are provided to reduce fragmentation and support cross-dataset training and fair comparison. Using NEBULA, we demonstrate that top-performing VLAs struggle with key capabilities such as spatial reasoning and dynamic adaptation, which are consistently obscured by conventional end-task success metrics. By measuring both what an agent can do and when it does so reliably, NEBULA provides a practical foundation for robust, general-purpose embodied agents.
SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of California, San Diego(加州大学圣地亚哥分校);Tel-Aviv University(特拉维夫大学);Reichman University(赖希曼大学);Stanford University(斯坦福大学)
本文提出 SEVA 基准,用约 9 万张不同稀疏度的人类草图评估视觉算法的识别与人类响应对齐程度,并考察草图生成模型,揭示模型与人类仍有明显差距。
VISOR: A Vision-Language Model-based Test Oracle for Testing Robots
Simula Research Laboratory(Simula研究实验室);Oslo Metropolitan University(奥斯陆理工大学);Mondragon University(蒙dragon大学);National Institute of Informatics(国立信息研究所)
$A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
South China University of Technology(华南理工大学);Shenzhen University(深圳大学);Pazhou Laboratory(琶洲实验室);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(教育部大数据与智能机器人重点实验室);Shenzhen Graduate School, Peking University(北京大学深圳研究生院);UMass Amherst(马萨诸塞大学阿默斯特分校);Cisco Research(思科研究院)