arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.16263cs.ROcs.AIcs.CV
收藏

NEBULA:我们是否正确评估视觉-语言-动作智能体?

NEBULA: Do We Evaluate Vision-Language-Action Agents Correctly?

Department of Computer & Data Sciences(计算机与数据科学系);Case Western Reserve University(凯斯西储大学)

Jierui Peng, Yanyan Zhang, Yicheng Duan, Tuo Liang, Vipin Chaudhary, Yu Yin

AI总结:

本文提出面向单臂操作的统一评估生态系统 NEBULA,通过细粒度能力测试与系统性压力测试的双轴协议、标准化 API 和聚合数据集,揭示顶级 VLA 在空间推理与动态适应上的不足。

摘要:

视觉-语言-动作(VLA)智能体的评估受到粗糙的终端任务成功指标的阻碍,这类指标既无法提供精确的技能诊断,也无法衡量其对真实世界扰动的鲁棒性。碎片化的数据格局进一步加剧了这一挑战,妨碍了可复现研究和通用模型的开发。为解决这些局限,我们提出 NEBULA,这是一个面向单臂操作的统一生态系统,可实现诊断式、可复现的评估。NEBULA 采用新颖的双轴评估协议,将用于精确技能诊断的细粒度能力测试与衡量鲁棒性的系统性压力测试相结合。其提供标准化 API 和大规模聚合数据集,以减少碎片化,并支持跨数据集训练与公平比较。借助 NEBULA,我们证明表现最佳的 VLA 在空间推理和动态适应等关键能力上存在困难,而这些能力始终被传统的终端任务成功指标所掩盖。通过同时衡量智能体能做什么以及何时能可靠地完成,NEBULA 为鲁棒、通用的具身智能体提供了实用基础。

英文摘要:

The evaluation of Vision-Language-Action (VLA) agents is hindered by the coarse, end-task success metric that fails to provide precise skill diagnosis or measure robustness to real-world perturbations. This challenge is exacerbated by a fragmented data landscape that impedes reproducible research and the development of generalist models. To address these limitations, we introduce NEBULA, a unified ecosystem for single-arm manipulation that enables diagnostic and reproducible evaluation. NEBULA features a novel dual-axis evaluation protocol that combines fine-grained capability tests for precise skill diagnosis with systematic stress tests that measure robustness. A standardized API and a large-scale, aggregated dataset are provided to reduce fragmentation and support cross-dataset training and fair comparison. Using NEBULA, we demonstrate that top-performing VLAs struggle with key capabilities such as spatial reasoning and dynamic adaptation, which are consistently obscured by conventional end-task success metrics. By measuring both what an agent can do and when it does so reliably, NEBULA provides a practical foundation for robust, general-purpose embodied agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑