关注细节，logits到真相：视觉感知注意力与logits增强以减轻LVLMs中的幻觉
Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
AI总结：
本文提出一种无需训练的注意力干预算法，通过增强任务相关token的注意力和改进解码过程，有效减少LVLMs中的幻觉问题。
摘要：
现有的大型视觉-语言模型（LVLMs）表现出不足的视觉注意力，导致幻觉。为缓解这一问题，一些先前的研究调整并放大了视觉注意力。这些方法存在一个局限性，即为所有视觉token提升注意力不可避免地会增加对任务无关token的注意力。为解决这一挑战，我们提出了一种无需训练的注意力干预算法，基于任务相关token通常表现出高视觉-文本相似性的论点，以增强任务相关token的注意力。具体而言，提取视觉-文本交叉注意力子矩阵，这些子矩阵代表视觉-文本相关性，以构建重加权矩阵以重新分配注意力。此外，为了增强视觉token的贡献，我们将视觉注意力值注入到束搜索解码中，以识别具有更高视觉注意力的解决方案。广泛的实验表明，该方法在主流LVLMs上显著减少了幻觉，同时保持了生成内容的准确性和连贯性。
英文摘要：
Existing Large Vision-Language Models (LVLMs) exhibit insufficient visual attention, leading to hallucinations. To alleviate this problem, some previous studies adjust and amplify visual attention. These methods present a limitation that boosting attention for all visual tokens inevitably increases attention to task irrelevant tokens. To tackle this challenge, we propose a training free attentional intervention algorithm to enhance the attention of task-relevant tokens based on the argument that task-relevant tokens generally demonstrate high visual-textual similarities. Specifically, the vision-text cross-attention submatrices, which represent visual-textual correlations, are extracted to construct the reweighting matrices to reallocate attention. Besides, to enhance the contribution of visual tokens, we inject visual attention values into the beam search decoding to identify solutions with higher visual attention. Extensive experiments demonstrate that this method significantly reduces hallucinations across mainstream LVLMs, while preserving the accuracy and coherence of generated content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
