arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.28385cs.ROcs.AI
收藏

RoboGaze: 通过结构化视觉-语言分析评估机器人世界模型

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

Ho Chi Minh City University of Science, Vietnam National University(越南国家科学大学胡志明市大学);VinRobotics;VinUniversity;German Research Center for AI (DFKI)(人工智能研究中心(DFKI));University of Stuttgart(斯图加特大学);Max Planck Research School for Intelligent Systems(智能系统马克斯·普朗克研究学校);Technische Universität Darmstadt(达姆施塔特技术大学)
Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen, Minh Le, Doanh L. Thien, Tuan A. Tran, Hoang H. Tran, Dung D. Le, Vu N. Duong, Khoa D. Doan, Ngan Le, Daniel So… 展开作者

Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen, Minh Le, Doanh L. Thien, Tuan A. Tran, Hoang H. Tran, Dung D. Le, Vu N. Duong, Khoa D. Doan, Ngan Le, Daniel Sonntag, An T. Le, Vien Anh Ngo, Duy M. H. Nguyen, Tran V. Nhiem

AI总结:

提出RoboGaze,一种无需训练的多智能体VLM框架,通过任务-场景对齐、维度专家路由和批评验证三阶段流水线,对机器人操作视频进行结构化可解释评估,显著提升描述F1和时序对齐性能。

摘要:

近期机器人世界模型的进展使得用于具身预测和规划的合成视频生成成为可能。然而,评估这些视频具有挑战性:视觉上逼真的输出常常违反物理定律、时间一致性或任务逻辑,而传统指标和单一视觉-语言模型(VLM)评判器无法泛化或提供精确的诊断价值。我们提出RoboGaze,一种无需训练的多智能体VLM框架,为生成的机器人操作视频提供结构化、可解释的评估。给定任务指令和视频,RoboGaze通过三阶段流水线运行:任务-场景对齐、维度特定专家路由和批评验证。它输出时间定位的故障报告,这些报告归类于一种新颖的6维度、30类型的机器人特定分类法。为了基准测试RoboGaze,我们引入了一个包含382个片段的人工验证数据集,涵盖模拟和真实世界的多视角操作。评估八个开源和专有VLM骨干网络,RoboGaze显著优于零样本基线,将描述F1提升高达+43点,时序对齐(F1 x IoU)提升高达+37点,缩小了约85%与人类上限的差距。此外,其批评验证器缓解了标准VLM的“狼来了”假阳性缺陷,将干净片段准确率从低于25%提升至超过80%。RoboGaze为机器人世界模型的严格评估提供了一种可扩展、高度可解释的诊断工具。

英文摘要:

Robot world models now generate synthetic manipulation videos, but evaluating them is hard: realistic-looking outputs often violate physical laws, temporal consistency or task logic, while conventional metrics and monolithic Vision-Language Model (VLM) judges return scores that do not say what failed. We present RoboGaze, a training-free multi-agent VLM evaluator that grounds the task and scene, routes each subtask to dimension-specific specialists and verifies their findings with a critic, producing temporally localized diagnostic reports that state what failed, when, why and how severely, under a schema of 6 dimensions and 30 failure types whose annotation reliability and coverage we measure. To benchmark it, we introduce RoboGazeBench, 914 human-annotated clips from five generators, conditioned on simulated and real-world data and running from 4.9 to 148.9 s. Across eight open-source and proprietary VLM backbones, RoboGaze outperforms every baseline; even a published agentic detector given the same schema and a slightly larger inference budget trails it by 10.1-12.3 description-F1 points, a margin that blinded human raters confirm. On the strongest backbone it closes 83% of the gap between chain-of-thought prompting and human agreement, and its critic curbs the "cry-wolf" false positives of standard VLMs, lifting mean clean-clip accuracy from 17% to 85%. The advantage is largest on held-out generators and long clips, and carries over to selecting generated demonstrations (80% vs. 71% precision).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑