面向多模态视觉语言模型中的有定位（Grounded）视觉空间推理
Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models
AI总结：
本研究针对大型视觉语言模型空间关系理解能力差的问题，证实目标定位能力不足是重要原因，提出结合名词短语grounding证据的细粒度组合式空间子句排序方法，并在代表性VLM上验证了效果。
摘要：
在大规模图文对数据集上训练以匹配图像与文本的大型视觉语言模型（VLM）已在多项视觉与语言任务中展现出出色的泛化能力。然而，近期多项研究表明这类模型缺乏细粒度理解能力，例如计数、识别动词、属性或关系的能力。本研究聚焦于空间关系理解能力的探究。此前该问题通过图文匹配（如Visual Spatial Reasoning基准）或视觉问答（如GQA或VQAv2）开展研究，两类方法均表现不佳，与人类表现存在较大差距。本研究通过可解释性工具进行定性分析、借助目标检测器开展定量分析，证实模型较差的目标定位（即“grounding， grounding”）能力是导致图文匹配表现不佳的一个影响因素。我们提出了一种替代的细粒度组合式方法，用于识别并排序空间子句，该方法结合与物体对应的名词短语及其位置的grounding证据，计算空间子句的最终排名。我们在代表性VLM（如LXMERT、GPV和MDETR）上验证了该方法，并对比、凸显了这些模型的空间关系推理能力。
英文摘要：
Large vision-and-language models (VLMs) trained to match images with text on large-scale datasets of image-text pairs have shown impressive generalization ability on several vision and language tasks. Several recent works, however, showed that these models lack fine-grained understanding, such as the ability to count and recognize verbs, attributes, or relationships. The focus of this work is to study the understanding of spatial relations. This has been tackled previously using image-text matching (e.g., Visual Spatial Reasoning benchmark) or visual question answering (e.g., GQA or VQAv2), both showing poor performance and a large gap compared to human performance. In this work, we show qualitatively (using explainability tools) and quantitatively (using object detectors) that the poor object localization "grounding" ability of the models is a contributing factor to the poor image-text matching performance. We propose an alternative fine-grained, compositional approach for recognizing and ranking spatial clauses that combines the evidence from grounding noun phrases corresponding to objects and their locations to compute the final rank of the spatial clause. We demonstrate the approach on representative VLMs (such as LXMERT, GPV, and MDETR) and compare and highlight their abilities to reason about spatial relationships.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
