CAPTURe：通过遮挡物体计数评估视觉语言模型的空间推理能力
CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting
AI总结：
提出新任务CAPTURe以评估视觉语言模型对遮挡物体的空间推理能力，发现主流VLMs在遮挡模式下计数表现显著下降，而人类几乎无误差。
摘要：
识别并对被遮挡（部分或完全隐藏）的物体进行推理对于理解视觉场景至关重要，因为遮挡在现实环境中频繁发生，并构成了空间理解的障碍。为了测试模型对多个被遮挡物体的推理能力，我们引入了一项新任务：通过未见区域对模式进行非模态计数（CAPTURe），该任务要求模型通过推断模式在遮挡物（阻挡场景部分区域的物体）背后的延续方式，来计数按模式排列的物体。CAPTURe 需要识别视觉模式和推理，这使其成为评估视觉语言模型（VLMs）是否理解遮挡模式并具备空间理解技能的有用测试平台。通过要求模型对被遮挡物体进行推理，CAPTURe 还测试了 VLMs 构建世界模型以填补缺失信息的能力。CAPTURe 包含两部分：（1）CAPTURe-real，包含手动筛选的真实物体模式图像；（2）CAPTURe-synthetic，包含生成模式图像的受控诊断集。我们在 CAPTURe 上评估了四个强大的 VLMs（GPT-4o、Intern-VL2、Molmo 和 Qwen2-VL），发现模型在遮挡和未遮挡模式的计数上都存在困难。关键的是，我们发现模型在遮挡情况下的表现更差，这表明 VLMs 在推断未见空间关系方面也存在不足：即使是像 GPT-4o 这样最强大的 VLMs 也无法在遮挡下进行计数。相比之下，我们发现人类在 CAPTURe 上的错误率极低。我们还发现，提供被遮挡物体位置的辅助信息能提高性能，这凸显了模型错误既源于无法处理遮挡，也源于在图像中计数的困难。代码和数据：https://github.com/atinpothiraj/CAPTURe
英文摘要：
Recognizing and reasoning about occluded (partially or fully hidden) objects is vital to understanding visual scenes, as occlusions frequently occur in real-world environments and act as obstacles for spatial comprehension. To test models' ability to reason about multiple occluded objects, we introduce a novel task, Counting Amodally for Patterns Through Unseen REgions (CAPTURe), which requires a model to count objects arranged in a pattern by inferring how the pattern continues behind an occluder (an object which blocks parts of the scene). CAPTURe requires both recognizing visual patterns and reasoning, making it a useful testbed for evaluating vision-language models (VLMs) on whether they understand occluded patterns and possess spatial understanding skills. By requiring models to reason about occluded objects, CAPTURe also tests VLMs' ability to form world models that would allow them to fill in missing information. CAPTURe consists of two parts: (1) CAPTURe-real, with manually filtered images of real objects in patterns and (2) CAPTURe-synthetic, a controlled diagnostic with generated patterned images. We evaluate four strong VLMs (GPT-4o, Intern-VL2, Molmo, and Qwen2-VL) on CAPTURe, finding that models struggle to count on both occluded and unoccluded patterns. Crucially, we find that models perform worse with occlusion, suggesting that VLMs are also deficient in inferring unseen spatial relationships: even the strongest VLMs like GPT-4o fail to count with occlusion. In contrast, we find that humans achieve very little error on CAPTURe. We also find that providing auxiliary information of occluded object locations increases performance, underscoring that the model error comes both from an inability to handle occlusion as well as difficulty in counting in images. Code and data: https://github.com/atinpothiraj/CAPTURe
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
