揭示并增强核心视觉区域：利用内部注意力动态缓解LVLMs中的幻觉
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs
AI总结：
PADE通过构建PAD图和系统令牌补偿，利用内部注意力动态提升LVLMs的视觉定位能力并减少幻觉。
摘要：
LVLMs在多模态推理能力方面取得了显著进展，但仍然容易产生幻觉，生成与视觉输入或用户指令不一致的输出。现有的无需训练的方法，包括对比解码和辅助专家模型，会导致几倍的计算开销，并可能引入潜在干扰，以及静态内部信号增强，往往容易受到注意力沉没现象的影响。我们发现，在注意力沉没的干扰下，LVLMs中的内部正注意力动态（PAD）自然揭示出语义核心的视觉区域。基于此，我们提出了正注意力动态增强（PADE），一种无需训练的注意力干预方法，通过构建PAD图来识别语义核心的视觉区域，应用每头中位绝对偏差缩放来适应性地控制干预强度，并利用系统令牌补偿来维持对复杂用户指令的关注并支持长期输出一致性。在多个LVLMs和基准测试中的实验表明，PADE提高了视觉定位并减少了幻觉，验证了利用内部注意力动态进行可靠多模态推理的有效性。
英文摘要：
LVLMs have achieved strong multimodal reasoning capabilities but remain prone to hallucinations, producing outputs inconsistent with visual inputs or user instructions. Existing training-free methods, including contrastive decoding and auxiliary expert models, which incur several times more computational overhead and may introduce potential interference, as well as static internal signal enhancement, are often vulnerable to the attention sink phenomenon. We find that internal Positive Attention Dynamics (PAD) in LVLMs naturally reveal semantically core visual regions under the distortions of attention sinks. Based on this, we propose Positive Attention Dynamics Enhancement (PADE), a training-free attention intervention that constructs a PAD map to identify semantically core visual regions, applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength, and leverages System-Token Compensation to maintain attention to complex user instructions and support long-term output consistency. Experiments on multiple LVLMs and benchmarks show that PADE improves visual grounding and reduces hallucinations, validating the effectiveness of leveraging internal attention dynamics for reliable multimodal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
