arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.19683cs.CV
收藏

从室内到开放世界:揭示MLLMs中的空间推理差距

From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs

University of Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院);Microsoft Research Asia(微软亚洲研究院)

Mingrui Wu, Zhaozhi Wang, Fangjinhua Wang, Jiaolong Yang, Marc Pollefeys, Tong Zhang

AI总结:

本文提出了一种大规模基准测试,通过户外数据集揭示MLLMs在空间推理方面的不足,并证明其依赖语言先验而非真实视觉推理。

摘要:

尽管多模态大语言模型(MLLMs)在语义任务上取得了令人印象深刻的表现,但其空间智能——对于稳健和 grounded 的人工智能系统至关重要——仍然发展不足。现有基准测试在诊断这一限制方面存在不足:它们要么专注于过于简化的定性推理,要么依赖于领域特定的室内数据,受缺乏具有可验证度量地面真实性的户外数据的限制。为了弥合这一差距,我们引入了一个大规模基准测试,该测试基于同步立体相机、LiDAR和IMU/GPS传感器捕获的行人视角视频。该数据集提供了具有度量精度的3D信息,使能够自动生成覆盖层次谱系的时空推理问题——从定性关系推理到定量度量和运动学理解。评估发现,在结构化的室内基准测试中观察到的性能提升在开放世界设置中消失。进一步使用合成异常场景和致盲测试的分析证实,当前的MLLMs严重依赖于语言先验而不是基于现实的视觉推理。因此,我们的基准测试提供了一个系统化的平台,用于诊断这些限制并推动物理上 grounded 的空间智能发展。

英文摘要:

While Multimodal Large Language Models (MLLMs) have achieved impressive performance on semantic tasks, their spatial intelligence--crucial for robust and grounded AI systems--remains underdeveloped. Existing benchmarks fall short of diagnosing this limitation: they either focus on overly simplified qualitative reasoning or rely on domain-specific indoor data, constrained by the lack of outdoor datasets with verifiable metric ground truth. To bridge this gap, we introduce a large-scale benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors. This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions that span a hierarchical spectrum--from qualitative relational reasoning to quantitative metric and kinematic understanding. Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings. Further analysis using synthetic abnormal scenes and blinding tests confirms that current MLLMs depend heavily on linguistic priors instead of grounded visual reasoning. Our benchmark thus provides a principled platform for diagnosing these limitations and advancing physically grounded spatial intelligence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑