arXiv 2509.078642025-11-17cs.CV

通过动态注意力定位追踪并缓解多模态大语言模型中的幻觉现象

Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization

MBZUAI；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析实验室）；King Abdullah University of Science and Technology（卡迪夫大学科学与技术大学）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）

Tiancheng Yang, Lin Zhang, Jiaye Lin, Guimin Hu, Di Wang, Lijie Hu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大语言模型的幻觉问题，提出动态注意力定位方法D-LEAF，通过LIAE和IAF诊断工具精准定位并修正错误注意力，在图像描述和视觉问答任务上显著提升性能，同时提供理论依据并保持效率。

摘要：

多模态大语言模型（MLLMs）在图像描述和视觉问答等任务上表现出色，但仍容易出现幻觉现象，即生成的文本与视觉输入冲突。先前的研究部分将此归因于视觉注意力不足，但现有的基于注意力的检测器和缓解方法通常在各层和注意力头之间应用统一调整，掩盖了错误的来源。在本文中，我们首先证明这些方法无法准确定位有问题的层。然后，我们引入两种诊断工具：标记异常层的层图像注意力熵（LIAE），以及对这些层内的注意力头进行评分的图像注意力焦点（IAF）。分析表明，LIAE能精准定位故障层，IAF能可靠地对需要修正的注意力头进行排序。在这些信号的引导下，我们提出动态分层熵与注意力融合（D-LEAF），这是一种任务无关、注意力引导的方法，可在推理过程中动态定位并修正错误，且开销可忽略不计。此外，通过建立D-LEAF与DPO的联系，我们为D-LEAF的有效性提供了理论依据。结果显示，我们的D-LEAF在标准图像描述基准上实现了53%的相对改进，在视觉问答任务上准确率和F1分数均提高约4%，在抑制幻觉的同时保持了效率。

英文摘要：

Multimodal Large Language Models (MLLMs) achieve strong performance on tasks like image captioning and visual question answering, but remain prone to hallucinations, where generated text conflicts with the visual input. Prior work links this partly to insufficient visual attention, but existing attention-based detectors and mitigation typically apply uniform adjustments across layers and heads, obscuring where errors originate. In this paper, we first show these methods fail to accurately localize problematic layers. Then, we introduce two diagnostics: Layer Image Attention Entropy (LIAE) which flags anomalous layers, and Image Attention Focus (IAF) which scores attention heads within those layers. Analysis shows that LIAE pinpoints faulty layers and IAF reliably ranks heads that warrant correction. Guided by these signals, we propose Dynamic Layer-wise Entropy and Attention Fusion (D-LEAF), a task-agnostic, attention-guided method that dynamically localizes and corrects errors during inference with negligible overhead. Furthermore, by establishing a connection between D-LEAF and DPO, we provide theoretical justification for the effectiveness of D-LEAF. Results show our D-LEAF delivers a 53\% relative improvement on standard captioning benchmarks, and on VQA both accuracy and F1-score improve by approximately 4\%, substantially suppressing hallucinations while preserving efficiency.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。