MINT：通过 Token 减少来缓解大型视觉语言模型中的幻觉
MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
AI总结：
提出免训练解码策略MINT，通过屏蔽不相关图像token和对比解码减少注意力冗余，引导模型聚焦关键视觉元素，有效缓解LVLM幻觉问题并提升视觉感知。
摘要：
幻觉一直是一个长期存在且不可避免的问题，阻碍了大型视觉语言模型（LVLMs）在需要高可靠性的领域中的应用。各种方法侧重于依赖数据标注或训练策略进行改进，但较少关注 LLM 的固有问题。为了填补这一空白，我们深入研究了 LVLM 解码过程中的注意力机制。有趣的是，我们的研究揭示了 LVLM 分层架构中普遍存在的注意力冗余现象，表现为深层中过度的图像处理以及过多非必要图像 token。基于这一观察，我们提出了 MINT，一种新颖的免训练解码策略，通过减少 token 来缓解幻觉。具体而言，我们通过屏蔽对不相关图像 token 的注意力，动态增强 LVLM 的局部感知能力。此外，我们使用对比解码，推动模型更加关注那些关键图像区域。我们的完整方法旨在引导模型在生成过程中更集中于关键视觉元素。在几个流行的公共基准上的大量实验结果表明，与原始模型相比，我们的方法在缓解由分散注意力引起的幻觉方面实现了 4% 的提升。同时，我们的方法被证明能够让模型多感知 5% 的视觉点，尽管我们减少了一组图像 token。
英文摘要：
Hallucination has been a long-standing and inevitable problem that hinders the application of Large Vision-Language Models (LVLMs) in domains that require high reliability. Various methods focus on improvement depending on data annotations or training strategies, yet place less emphasis on LLM's inherent problems. To fill this gap, we delve into the attention mechanism of the decoding process in the LVLM. Intriguingly, our investigation uncovers the prevalent attention redundancy within the hierarchical architecture of the LVLM, manifesting as overextended image processing in deep layers and an overabundance of non-essential image tokens. Stemming from the observation, we thus propose MINT, a novel training-free decoding strategy, MItigating hallucinations via tokeN reducTion. Specifically, we dynamically intensify the LVLM's local perception capability by masking its attention to irrelevant image tokens. In addition, we use contrastive decoding that pushes the model to focus more on those key image regions. Our full method aims to guide the model in concentrating more on key visual elements during generation. Extensive experimental results on several popular public benchmarks show that our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models. Meanwhile, our approach is demonstrated to make the model perceive 5% more visual points even though we reduce a suite of image tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
