arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.02537cs.CLcs.CVcs.LG
收藏

TopViewRS:作为俯视空间推理者的视觉-语言模型

TopViewRS: Vision-Language Models as Top-View Spatial Reasoners

University of Cambridge(剑桥大学)

Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, Ivan Vulić

AI总结:

本文提出 TopViewRS 数据集以评估视觉-语言模型的俯视空间推理能力,发现其与人类差距超50%,即使思维链推理也仅有限提升表现,凸显了增强该能力的迫切需求。

摘要:

俯视视角指代人类阅读和推理不同类型地图的一种典型方式,它对于人类以及由大型视觉-语言模型(VLMs)支持的“非人类”智能体的定位和导航至关重要。然而,现代 VLMs 的空间推理能力仍未得到验证和充分探索。因此,在本研究中,我们探讨了它们从俯视视角理解和推理空间关系的能力。对俯视视角的关注还使得能够在不同粒度的空间推理上进行受控评估;我们明确区分了不同的能力(例如,识别特定物体与理解它们的相对位置)。我们引入了 TopViewRS(Top-View Reasoning in Space)数据集,包含 11,384 道选择题,以真实或语义俯视图作为视觉输入。随后,我们利用该数据集在具有不同复杂程度的 4 项感知和推理任务上研究和评估 VLMs。对 10 个具有代表性的开源和闭源 VLMs 的评估显示,其与人类平均表现之间存在超过 50% 的差距,在某些情况下甚至低于随机基线。尽管进一步的实验表明,思维链(Chain-of-Thought)推理平均可将模型能力提升 5.82%,但 VLMs 的整体表现依然有限。我们的研究结果强调了增强模型在俯视空间推理方面能力的迫切需求,并为未来在现实世界多模态任务中实现 VLMs 达到人类水平的研究奠定了基础。

英文摘要:

Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of `non-human' agents, such as the ones backed by large Vision-Language Models (VLMs). Nonetheless, spatial reasoning capabilities of modern VLMs remain unattested and underexplored. In this work, we thus study their capability to understand and reason over spatial relations from the top view. The focus on top view also enables controlled evaluations at different granularity of spatial reasoning; we clearly disentangle different abilities (e.g., recognizing particular objects versus understanding their relative positions). We introduce the TopViewRS (Top-View Reasoning in Space) dataset, consisting of 11,384 multiple-choice questions with either realistic or semantic top-view map as visual input. We then use it to study and evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity. Evaluation of 10 representative open- and closed-source VLMs reveals the gap of more than 50% compared to average human performance, and it is even lower than the random baseline in some cases. Although additional experiments show that Chain-of-Thought reasoning can boost model capabilities by 5.82% on average, the overall performance of VLMs remains limited. Our findings underscore the critical need for enhanced model capability in top-view spatial reasoning and set a foundation for further research towards human-level proficiency of VLMs in real-world multimodal tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑