学习定位物体可提升视觉大语言模型(V-LLMs)的空间推理能力
Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs
AI总结:
针对现有视觉大语言模型空间推理与定位能力弱的问题,本研究通过基于图像空间坐标的指令微调等策略提升其空间感知能力,在多任务多数据集上实现了性能提升。
摘要:
大语言模型(LLMs)被整合到视觉领域任务中催生了视觉大语言模型(V-LLMs),这类模型在视觉语言任务尤其是视觉问答(VQA)中取得了出色表现。但现有V-LLMs(如BLIP-2、LLaVA)的空间推理与定位感知能力较弱,尽管能生成描述性强、内容详尽的文本答案,却无法完成区分左右位置这类简单任务。本研究探索了基于图像空间坐标的指令微调目标如何为V-LLMs注入空间感知能力,找到了能提升V-LLMs空间感知能力的最优坐标表示、数据高效型指令微调目标以及伪数据生成策略。此外,我们得到的模型提升了图像和视频领域的VQA效果,减少了不良幻觉,还能生成更优质的上下文物体描述。在涵盖14个不同数据集的5项视觉语言任务上开展的实验证明,我们提出的框架实现了显著的性能提升。
英文摘要:
Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA). However, existing V-LLMs (e.g. BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness. Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location. In this work, we explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs. We discover optimal coordinate representations, data-efficient instruction fine-tuning objectives, and pseudo-data generation strategies that lead to improved spatial awareness in V-LLMs. Additionally, our resulting model improves VQA across image and video domains, reduces undesired hallucination, and generates better contextual object descriptions. Experiments across 5 vision-language tasks involving 14 different datasets establish the clear performance improvements achieved by our proposed framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。