告诉模型该看哪里：通过视觉引导注意力缓解大语言模型的幻觉
Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention
AI总结：
本文提出视觉引导注意力（VGA），通过构建精确的视觉基础并引导模型关注相关区域，缓解大语言模型在视觉任务中的幻觉问题，且无需额外训练。
摘要：
视觉注意力是大语言模型（MLLMs）解释视觉信息的主要机制；然而，其有限的定位能力常导致幻觉。我们观察到，尽管MLLMs能准确从视觉标记中提取视觉语义，但在后续推理中未能充分利用这一优势。为此，我们提出了视觉引导注意力（VGA），一种无需训练的方法，首先通过利用视觉标记的语义内容构建精确的视觉基础，然后使用此基础引导模型的注意力聚焦于相关视觉区域。在图像描述任务中，VGA通过在生成过程中动态抑制已描述的区域进一步细化此引导。在VGA中，每个标记仅经历一次前向传递，引入了可忽略的延迟开销。此外，VGA与高效的注意力实现（如FlashAttention）完全兼容。在多种MLLMs和多个幻觉基准测试中，VGA实现了最先进的去幻觉性能。进一步分析证实，显式的视觉引导在增强MLLMs的视觉理解能力方面起着关键作用。
英文摘要：
Visual attention serves as the primary mechanism through which MLLMs interpret visual information; however, its limited localization capability often leads to hallucinations. We observe that although MLLMs can accurately extract visual semantics from visual tokens, they fail to fully leverage this advantage during subsequent inference. To address this limitation, we propose Vision-Guided Attention (VGA), a training-free method that first constructs precise visual grounding by exploiting the semantic content of visual tokens, and then uses this grounding to guide the model's focus toward relevant visual regions. In image captioning, VGA further refines this guidance dynamically during generation by suppressing regions that have already been described. In VGA, each token undergoes only a single forward pass, introducing a negligible latency overhead. In addition, VGA is fully compatible with efficient attention implementations such as FlashAttention. Extensive experiments across diverse MLLMs and multiple hallucination benchmarks demonstrate that VGA achieves state-of-the-art dehallucination performance. Further analysis confirms that explicit visual guidance plays a crucial role in enhancing the visual understanding capabilities of MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
