arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01640cs.CL
收藏

A2Eval: 基于代理的自动评估用于具身脑

A2Eval: Agentic and Automated Evaluation for Embodied Brain

Zhejiang University, China(浙江大学);Westlake University, China(西湖大学);Beijing Innovation Center of Humanoid Robotics, China(北京人形机器人创新中心);University of Manchester, UK(曼彻斯特大学);Southern University of Science(南方科技大学);Xiamen University Malaysia, Malaysia(厦门大学马来西亚分校)

Shuai Zhang, Jiayu Hu, Zijie Chen, Zeyuan Ding, Yi Zhang, Yingji Zhang, Ziyi Zhou, Junwei Liao, Shengjie Zhou, Yong Dai, Zhenzhong Lan, Xiaozhu Ju

AI总结:

A2Eval提出首个基于代理的自动评估框架,通过两个协作代理自动化基准编纂和评估,显著提升评估效率并减少成本,同时保持高保真度。

摘要:

当前具身视觉语言模型(VLM)的评估依赖于静态的、由专家定义的、手动标注的基准,这些基准存在严重的冗余性和覆盖不平衡问题。这种劳动密集型的范式消耗计算和标注资源,增加成本,并扭曲模型排名,最终抑制了迭代发展。为了解决这一问题,我们提出了Agentic Automatic Evaluation(A2Eval),这是首个基于代理的框架,通过两个协作代理自动化基准的编纂和评估。数据代理自主诱导能力维度,并组装一个平衡、紧凑的评估套件,而评估代理合成并验证可执行的评估流程,使评估能够完全自动、高保真度地进行。在10个基准和13个模型上评估,A2Eval将评估套件压缩了85%,降低了总计算成本77%,在保持评估质量的同时实现了4.6倍的速度提升。关键的是,A2Eval纠正了系统性的排名偏差,提高了人类一致性到Spearman的rho=0.85,并保持了高排名保真度(Kendall的tau=0.81),建立了高保真、低成本具身评估的新标准。我们的代码和数据将很快公开。

英文摘要:

Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑