仅在需要时看：用于缓解LVLM中幻觉的上下文感知注意力干预
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
AI总结：
提出训练自由的上下文感知注意力干预（CAI），通过两轴选择性（看哪里和何时干预）在解码时针对性增强视觉 grounding，有效缓解物体幻觉并保持语言流畅性。
摘要：
大型视觉语言模型（LVLM）在多模态任务中表现出色，但仍容易产生物体幻觉。先前的训练自由补救方法通常均匀增强视觉信号，这可能会放大无关区域并引入虚假证据，损害流畅性。我们提出上下文感知注意力干预（CAI），一种训练自由的推理时机制，通过两轴选择性（看哪里和何时干预）强制执行“仅在需要时看”原则。在每个解码步骤，CAI从早期层表示中导出令牌特定的视觉相关性以定位语义对齐区域，并对深层中视觉接地退化且不确定性突增的令牌应用保守的、熵和深度门控的注意力倾斜，而自信令牌和无关区域基本保持不变。这种针对性干预增强了视觉接地，同时保持了语言流畅性，即使没有对比解码（作为可选的辅助偏置抑制模块）也能取得一致改进。在多个LVLM骨干和基准上的广泛实验表明，CAI实现了最先进的幻觉缓解效果，我们的分析将CAI描述为一种KL最小化注意力重加权，在非激活门控或小倾斜下具有有界干扰。代码可在https://github.com/Iris1946/CAI获取。
英文摘要：
Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
