MDSAM：用于减轻LVLMs幻觉的记忆驱动稀疏注意力矩阵
MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation
AI总结：
提出无需训练的MDSAM方法，通过记忆并动态细化各层图像令牌的注意力，有效减少LVLMs幻觉，提升图像描述和视觉问答的可靠性。
摘要：
大型视觉语言模型（LVLMs）中的幻觉通常源于模型在解码过程中对图像令牌的敏感性，这体现在生成真实实体和幻觉实体时观察到的注意力峰值上。为了解决这个问题，我们提出了记忆驱动稀疏注意力矩阵（MDSAM），一种新颖的无需训练的方法，它动态捕获并细化每一层分配给图像令牌的注意力。MDSAM记忆注意力模式，并通过解码过程中的对齐激活更新，增强对相关图像令牌的关注，同时有效减少幻觉。我们在多个基准上评估了MDSAM，用于图像描述和视觉问答等任务，证明其能够持续减少幻觉并提高可靠性。MDSAM兼容各种LVLM架构，突显了其在无需额外训练或外部工具的情况下减轻幻觉的适应性和有效性。
英文摘要：
Hallucinations in large vision-language models (LVLMs) often stem from the model's sensitivity to image tokens during decoding, as evidenced by attention peaks observed when generating both real and hallucinated entities. To address this, we propose Memory-Driven Sparse Attention Matrix (MDSAM) , a novel training-free approach that dynamically captures and refines the attention allocated to image tokens at each layer. MDSAM memorizes attention patterns and activates updates through alignment during decoding, enhancing focus on relevant image tokens while effectively reducing hallucinations. We evaluate MDSAM on multiple benchmarks for tasks such as image captioning and visual question answering, demonstrating its ability to consistently reduce hallucinations and improve reliability. Compatible with various LVLM architectures, MDSAM highlights its adaptability and effectiveness in mitigating hallucinations without requiring additional training or external tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
