你的大型视觉-语言模型仅需少量注意力头即可实现视觉定位
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
AI总结：
本文发现冻结的大型视觉-语言模型中仅少数注意力头即可实现视觉定位，并提出一种仅用三个定位头的免训练框架，取得与需微调方法相当的定位性能。
摘要：
视觉定位旨在定位与自由形式文本描述相对应的图像区域。近年来，大型视觉-语言模型（LVLMs）强大的多模态能力推动了视觉定位的显著进步，但这些方法不可避免地需要微调和额外的模型组件来显式生成边界框或分割掩码。然而，我们发现冻结的LVLMs中少数几个注意力头展现出强大的视觉定位能力。我们将这些能够持续捕获与文本语义相关的物体位置的注意力头称为定位头。利用定位头，我们提出了一种简单有效的免训练视觉定位框架，该框架利用来自定位头的文本到图像注意力图来识别目标物体。令人惊讶的是，在数千个注意力头中仅需三个就足以取得与现有基于LVLM且需要微调的视觉定位方法相比具有竞争力的定位性能。我们的发现表明，LVLMs能够基于对文本-图像关系的深层理解天然地定位物体，因为它们会隐式地关注相关图像区域以生成信息丰富的文本输出。所有源代码将向公众开放。
英文摘要：
Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements in visual grounding, though they inevitably require fine-tuning and additional model components to explicitly generate bounding boxes or segmentation masks. However, we discover that a few attention heads in frozen LVLMs demonstrate strong visual grounding capabilities. We refer to these heads, which consistently capture object locations related to text semantics, as localization heads. Using localization heads, we introduce a straightforward and effective training-free visual grounding framework that utilizes text-to-image attention maps from localization heads to identify the target objects. Surprisingly, only three out of thousands of attention heads are sufficient to achieve competitive localization performance compared to existing LVLM-based visual grounding methods that require fine-tuning. Our findings suggest that LVLMs can innately ground objects based on a deep comprehension of the text-image relationship, as they implicitly focus on relevant image regions to generate informative text outputs. All the source codes will be made available to the public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
