指代表达式作为视觉-语言模型中空间语言接地的透镜
Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models
AI总结：
本文提出用指代表达式理解任务评估视觉-语言模型的空间推理能力，分析其在目标歧义、复杂空间表达和否定表达下的表现，揭示不同模型在各类空间语义上的优势与不足。
摘要：
空间推理是人类认知的重要组成部分，也是最新视觉-语言模型（VLMs）表现出困难的一个领域。目前的分析工作使用图像描述任务和视觉问答。在这项工作中，我们提出改用指代表达式理解（Referring Expression Comprehension）任务作为评估视觉-语言模型空间推理能力的平台。该平台为更深入地分析空间理解和接地能力提供了机会，当存在以下情况时：1）目标检测中存在歧义，2）具有较长句子结构和多个空间关系的复杂空间表达，以及3）带有否定（“不”）的表达。在我们的分析中，我们使用任务特定架构以及大型视觉-语言模型，并突出它们在处理这些特定情况时的优势和劣势。虽然所有这些模型都面临当前任务的挑战，但相对行为取决于底层模型和空间语义的具体类别（拓扑、方向、邻近等）。我们的结果突出了这些挑战和行为，并为研究空白和未来方向提供了见解。
英文摘要：
Spatial Reasoning is an important component of human cognition and is an area in which the latest Vision-language models (VLMs) show signs of difficulty. The current analysis works use image captioning tasks and visual question answering. In this work, we propose using the Referring Expression Comprehension task instead as a platform for the evaluation of spatial reasoning by VLMs. This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not'). In our analysis, we use task-specific architectures as well as large VLMs and highlight their strengths and weaknesses in dealing with these specific situations. While all these models face challenges with the task at hand, the relative behaviors depend on the underlying models and the specific categories of spatial semantics (topological, directional, proximal, etc.). Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
