MHSA：一种轻量级框架，通过引导注意力缓解LVLMs中的幻觉
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
AI总结：
本文提出MHSA框架，通过学习修正跨模态注意力模式来缓解LVLMs中的幻觉，采用简单三层MLP生成器和DHCP判别器信号指导训练，无需修改模型参数即可在多种数据集和模型上有效减少判别和生成幻觉。
摘要：
大型视觉-语言模型（LVLMs）在多样化的多模态任务中取得了显著性能，但仍然存在幻觉问题，即生成与视觉输入不一致的内容。先前工作DHCP（通过跨模态注意力模式检测幻觉）从跨模态注意力的角度探索了幻觉检测，但未解决幻觉缓解问题。本文提出MHSA（通过引导注意力缓解幻觉），一种轻量级框架，通过学习修正LVLMs中的跨模态注意力模式来缓解幻觉。MHSA训练一个简单的三层MLP生成器生成修正的注意力，由DHCP判别器和LVLM本身的监督信号指导。在推理过程中，MHSA通过简单地将原始跨模态注意力替换为修正后的注意力，即可在各种数据集和LVLMs上缓解判别和生成幻觉，而无需修改任何LVLM参数。通过将跨模态注意力机制从幻觉检测扩展到幻觉缓解，MHSA为LVLMs中的幻觉研究提供了新的视角，并有助于提高其可靠性。
英文摘要：
Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that is inconsistent with the visual input. Prior work DHCP (Detecting Hallucinations by Cross-modal Attention Pattern) has explored hallucination detection from the perspective of cross-modal attention, but does not address hallucination mitigation. In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that mitigates hallucinations by learning to correct cross-modal attention patterns in LVLMs. MHSA trains a simple three-layer MLP generator to produce corrected attention, guided by supervisory signals from the DHCP discriminator and the LVLM itself. During inference, MHSA mitigates both discriminative and generative hallucinations across various datasets and LVLMs by simply replacing the original cross-modal attention with the corrected one, without modifying any LVLM parameters. By extending cross-modal attention mechanisms from hallucination detection to hallucination mitigation, MHSA offers a novel perspective on hallucination research in LVLMs and helps enhance their reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
