VEGAS: 通过视觉编码器注意力引导的自适应转向来缓解大视觉-语言模型中的幻觉
VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
AI总结：
VEGAS通过整合视觉编码器注意力图并自适应引导标记，有效缓解大视觉-语言模型中的幻觉问题。
摘要：
大型视觉-语言模型（LVLMs）表现出在联合推理视觉和文本输入方面的出色能力。然而，它们经常产生在语言上流畅但与视觉证据事实不一致的输出，即产生幻觉。尽管已有努力减轻此类幻觉，但一个关键问题仍然存在：哪种形式的视觉注意力可以在解码过程中有效抑制幻觉？在本工作中，我们提供了一个简单的答案：视觉编码器自身的注意力图。我们证明，当最终的视觉-注意力图未能集中在关键图像对象上时，LVLMs倾向于产生幻觉，而视觉编码器更集中的注意力图显著减少了幻觉。为进一步研究原因，我们分析了解码过程中视觉-文本冲突，并发现这些冲突在语言模型的中间层达到峰值。将视觉编码器的注意力图注入这些层可以有效抑制幻觉。基于这些见解，我们引入了VEGAS，一种简单而有效的推理时间方法，它将视觉编码器的注意力图整合到语言模型的中间层，并自适应地引导未能集中在关键图像对象上的标记。在多个基准上的广泛实验表明，VEGAS在减少幻觉方面始终取得最先进的性能。
英文摘要：
Large vision-language models (LVLMs) exhibit impressive ability to jointly reason over visual and textual inputs. However, they often produce outputs that are linguistically fluent but factually inconsistent with the visual evidence, i.e., they hallucinate. Despite growing efforts to mitigate such hallucinations, a key question remains: what form of visual attention can effectively suppress hallucinations during decoding? In this work, we provide a simple answer: the vision encoder's own attention map. We show that LVLMs tend to hallucinate when their final visual-attention maps fail to concentrate on key image objects, whereas the vision encoder's more concentrated attention maps substantially reduce hallucinations. To further investigate the cause, we analyze vision-text conflicts during decoding and find that these conflicts peak in the language model's middle layers. Injecting the vision encoder's attention maps into these layers effectively suppresses hallucinations. Building on these insights, we introduce VEGAS, a simple yet effective inference-time method that integrates the vision encoder's attention maps into the language model's mid-layers and adaptively steers tokens which fail to concentrate on key image objects. Extensive experiments across multiple benchmarks demonstrate that VEGAS consistently achieves state-of-the-art performance in reducing hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
