arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.00799cs.CVcs.CLcs.LG
收藏

多模态语言模型无法发现空间不一致性

Multimodal Language Models Cannot Spot Spatial Inconsistencies

University of California, Davis(加州大学戴维斯分校);Shell(壳牌);TU Delft(代尔夫特理工大学)

Om Khangaonkar, Hadi J. Rad, Hamed Pirsiavash

AI总结:

研究探讨多模态大语言模型在识别3D几何空间不一致性方面的不足,提出生成空间不一致图像对的方法,发现先进模型在该任务上表现欠佳,揭示其对物理世界理解的脆弱性。

摘要:

空间一致性是视觉世界的基本属性,也是旨在理解物理现实的模型的关键要求。尽管近期取得进展,多模态大语言模型(MLLMs)在跨多视角的3D几何推理上仍面临挑战。我们引入更具挑战性的任务:给定同一场景的两个视角,识别违反3D运动一致性的物体。我们提出一种简单且可扩展的方法,生成真实且空间不一致的图像对,从而系统评估该能力。结果表明,最先进的MLLMs在该任务上显著逊色于人类观察者,并在不同场景属性下表现出显著的可变性,揭示了其对3D结构理解的脆弱性和不完整性。我们希望本研究的结果强调了开发更深入理解物理世界方法的必要性。

英文摘要:

Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances, multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑