带参考对象的推理路径激发大视觉语言模型的定量空间推理能力
Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
AI总结:
该研究针对大视觉语言模型定量空间推理能力不足的问题,构建了Q-Spatial Bench基准,提出零样本提示技术SpatialPrompt,借助参考对象推理路径显著提升了多款主流VLM的空间推理成功率且无需微调。
摘要:
尽管近期研究已展现视觉语言模型(VLM)借助自然语言描述图像中复杂关系的能力,但它们针对物体尺寸与距离开展定量推理的能力仍未得到充分探索。本文提出了一个人工标注基准Q-Spatial Bench,涵盖五大类共271个问题,专为定量空间推理任务设计,并系统探究了当前最先进(SoTA)的VLM在该任务上的表现。分析显示,物体间距离推理对SoTA VLM而言难度极高;不过不同VLM的表现差异显著,表现最优的两个模型之间存在超过40分的差距。研究还意外发现,当回复中自然出现使用参考对象的推理路径时,表现最佳的VLM的成功率可提升19个百分点。受此启发,本文开发了零样本提示技术SpatialPrompt,引导VLM将参考对象作为视觉线索来回答定量空间问题。通过SpatialPrompt指示VLM在推理路径中使用参考对象,Gemini 1.5 Pro、Gemini 1.5 Flash和GPT-4V的成功率分别提升了40、20和30个百分点以上。需要强调的是,这些显著提升无需额外数据、模型架构修改或微调即可实现。
英文摘要:
Despite recent advances demonstrating vision-language models' (VLMs) abilities to describe complex relationships in images using natural language, their capability to quantitatively reason about object sizes and distances remains underexplored. In this work, we introduce a manually annotated benchmark, Q-Spatial Bench, with 271 questions across five categories designed for quantitative spatial reasoning and systematically investigate the performance of state-of-the-art VLMs on this task. Our analysis reveals that reasoning about distances between objects is particularly challenging for SoTA VLMs; however, some VLMs significantly outperform others, with an over 40-point gap between the two best performing models. We also make the surprising observation that the success rate of the top-performing VLM increases by 19 points when a reasoning path using a reference object emerges naturally in the response. Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues. By instructing VLMs to use reference objects in their reasoning paths via SpatialPrompt, Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V improve their success rates by over 40, 20, and 30 points, respectively. We emphasize that these significant improvements are obtained without needing more data, model architectural modifications, or fine-tuning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。