通过区域感知注意力重校准减轻视觉语言模型中的对象幻觉
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
AI总结：
提出一种无需训练的区域感知自适应加权机制，通过计算注意力头的稳健统计中点并利用跨头分歧动态调整干预预算，以连续惩罚调制抑制幻觉路径，有效纠正视觉语义错位，同时保持生成流畅性。
摘要：
生成事实上不正确的对象（通常称为对象幻觉）仍然是大型视觉语言模型（LVLMs）中的一个持久挑战。当前解决该问题的方法——从昂贵的数据驱动微调和延迟较高的对比解码到刚性的注意力头截断——常常在计算效率或模型特征空间的连续性上做出妥协。为克服这些限制，我们引入了一种新颖的、无需训练的推理策略，该策略作为一种区域感知的自适应加权机制，动态纠正语义漂移，而不依赖于突然的启发式截断。通过计算各注意力头上的离群值稳健统计中点，我们为可靠的视觉表示建立了一个稳定锚点。然后，我们利用跨区域映射的跨头分歧来动态确定干预预算，通过连续惩罚调制温和地抑制引起幻觉的注意力路径。这种重校准过程有效纠正了视觉语义错位，同时完全保留了生成流畅性和语言先验。在包括CHAIR、POPE和MME在内的标准多模态基准上的全面评估表明，我们的策略显著减少了实例级和句子级幻觉。结果展示了与当代基线相比的最先进性能，证实了我们方法的效率和算法鲁棒性。我们的代码将公开。
英文摘要：
The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
