RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis
Ho Chi Minh City University of Science, Vietnam National University(越南国家科学大学胡志明市大学);VinRobotics;VinUniversity;German Research Center for AI (DFKI)(人工智能研究中心(DFKI));University of Stuttgart(斯图加特大学);Max Planck Research School for Intelligent Systems(智能系统马克斯·普朗克研究学校);Technische Universität Darmstadt(达姆施塔特技术大学)
Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen, Minh Le, Doanh L. Thien, Tuan A. Tran, Hoang H. Tran, Dung D. Le, Vu N. Duong, Khoa D. Doan, Ngan Le, Daniel So… 展开作者
Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen, Minh Le, Doanh L. Thien, Tuan A. Tran, Hoang H. Tran, Dung D. Le, Vu N. Duong, Khoa D. Doan, Ngan Le, Daniel Sonntag, An T. Le, Vien Anh Ngo, Duy M. H. Nguyen, Tran V. Nhiem
近期机器人世界模型的进展使得用于具身预测和规划的合成视频生成成为可能。然而,评估这些视频具有挑战性:视觉上逼真的输出常常违反物理定律、时间一致性或任务逻辑,而传统指标和单一视觉-语言模型(VLM)评判器无法泛化或提供精确的诊断价值。我们提出RoboGaze,一种无需训练的多智能体VLM框架,为生成的机器人操作视频提供结构化、可解释的评估。给定任务指令和视频,RoboGaze通过三阶段流水线运行:任务-场景对齐、维度特定专家路由和批评验证。它输出时间定位的故障报告,这些报告归类于一种新颖的6维度、30类型的机器人特定分类法。为了基准测试RoboGaze,我们引入了一个包含382个片段的人工验证数据集,涵盖模拟和真实世界的多视角操作。评估八个开源和专有VLM骨干网络,RoboGaze显著优于零样本基线,将描述F1提升高达+43点,时序对齐(F1 x IoU)提升高达+37点,缩小了约85%与人类上限的差距。此外,其批评验证器缓解了标准VLM的“狼来了”假阳性缺陷,将干净片段准确率从低于25%提升至超过80%。RoboGaze为机器人世界模型的严格评估提供了一种可扩展、高度可解释的诊断工具。
英文摘要:
Robot world models now generate synthetic manipulation videos, but evaluating them is hard: realistic-looking outputs often violate physical laws, temporal consistency or task logic, while conventional metrics and monolithic Vision-Language Model (VLM) judges return scores that do not say what failed. We present RoboGaze, a training-free multi-agent VLM evaluator that grounds the task and scene, routes each subtask to dimension-specific specialists and verifies their findings with a critic, producing temporally localized diagnostic reports that state what failed, when, why and how severely, under a schema of 6 dimensions and 30 failure types whose annotation reliability and coverage we measure. To benchmark it, we introduce RoboGazeBench, 914 human-annotated clips from five generators, conditioned on simulated and real-world data and running from 4.9 to 148.9 s. Across eight open-source and proprietary VLM backbones, RoboGaze outperforms every baseline; even a published agentic detector given the same schema and a slightly larger inference budget trails it by 10.1-12.3 description-F1 points, a margin that blinded human raters confirm. On the strongest backbone it closes 83% of the gap between chain-of-thought prompting and human agreement, and its critic curbs the "cry-wolf" false positives of standard VLMs, lifting mean clean-clip accuracy from 17% to 85%. The advantage is largest on held-out generators and long clips, and carries over to selecting generated demonstrations (80% vs. 71% precision).
SKA-Bench: A Fine-Grained Benchmark for Evaluating Structured Knowledge Understanding of LLMs
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团)
University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);Emory University(埃默里大学);University of Minnesota(明尼苏达大学);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校);UMass Chan Medical School(UMass Chan医学学院)
StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text
Fudan University(复旦大学);Shanghai Key Laboratory of Data Science(上海市数据科学重点实验室);School of Computer Science(计算机科学学院);Fudan-Aishu Cognitive Intelligence Joint Research Center(复旦-阿里智能认知联合研究中心)
Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages
Department of Radiology and Nuclear Medicine, Erasmus MC Cancer Institute, University Medical Center Rotterdam, the Netherlands(荷兰埃因霍温麦斯特大学医学中心放射科和核医学科,埃因霍温麦斯特癌症研究所);Department of Pathology and Clinical Bioinformatics, Erasmus MC Cancer Institute, University Medical Center Rotterdam, the Netherlands(荷兰埃因霍温麦斯特大学医学中心病理学与临床生物信息学科,埃因霍温麦斯特癌症研究所);Faculty of Informatics, Masaryk University(马萨里克大学信息学院);Institute of Computer Science, Masaryk University(马萨里克大学计算机科学研究所);Department of Medical Physics, Leeds Teaching Hospitals NHS Trust, UK(英国利兹教学医院国家卫生服务信托基金,利兹大学生物医学研究中心);Leeds Biomedical Research Centre, University of Leeds, UK(利兹大学生物医学成像科,利兹,英国);Biomedical Imaging, University of Leeds, Leeds, United Kingdom(荷兰埃因霍温麦斯特大学医学中心外科肿瘤学和胃肠外科科,埃因霍温麦斯特癌症研究所);Department of Surgical Oncology and Gastrointestinal Surgery, Erasmus MC Cancer Institute, University Medical Center Rotterdam, the Netherlands(奥地利格拉茨BBMRI-ERIC机构);BBMRI-ERIC, Graz, Austria(荷兰埃因霍温麦斯特大学医学中心流行病学系,罗特万,荷兰);Department of Epidemiology, Erasmus MC University Medical Center, Rotterdam, the Netherlands(马萨里克纪念癌症研究所生物信息学研究组);Bioinformatics Research Group, Masaryk Memorial Cancer Institute
ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);Health Data Research UK(英国健康数据研究所);School of Computer Science, Peking University(北京大学计算机学院);National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);ETH Zurich(苏黎世联邦理工学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The 63rd Research Institute, National University of Defense Technology(国防科技大学第六十三研究所);Nanjing University of Aeronautics and Astronautics(南京航空航天大学)