通过注意力不平衡修正减轻LVLM中的对象幻觉
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
AI总结：
发现多模态和token间注意力不平衡是对象幻觉的因果因素，提出轻量级解码干预方法AIR，通过重新分配注意力权重修正不平衡，在多个基准上减少幻觉达35.1%，并提升通用能力。
摘要：
大型视觉-语言模型（LVLMs）中的对象幻觉严重损害了其在现实应用中的可靠性，对它们在自动驾驶和医学图像分析等高风险场景中的部署构成了关键障碍。通过系统的实证研究，我们发现跨模态（即视觉和语言）和模态内（单个token之间）的不平衡注意力分配与对象幻觉的发生存在强因果相关性。利用这一洞察，我们引入了一个新概念——注意力不平衡，它不仅量化了注意力差异的程度，还直观地描绘了驱动对象幻觉的潜在模式（例如，对无关语言token的过度关注或对判别性视觉特征的关注不足）。为了减轻对象幻觉，我们进一步提出了注意力不平衡修正（AIR），这是一种轻量级的解码时干预方法，通过重新分配注意力权重和调整注意力分布来修正模态级和token级的不平衡。在四个主流LVLM和三个基准（CHAIR、POPE和MM-Vet）上，与七个基线进行的大量评估表明，AIR持续降低对象幻觉率，与基线相比最高减少35.1%，同时在多种视觉-语言任务中提升LVLMs的通用能力高达15.9%。
英文摘要：
Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis. Through systematic empirical investigation, we identify that the imbalanced attention allocation, both across modalities (i.e., vision and language) and within modalities (among individual tokens), exhibits a strong causal correlation with the occurrence of object hallucination. Leveraging this insight, we introduce a novel concept termed attention imbalance, which not only quantifies the degree of attention disparity but also visually delineates the underlying patterns (e.g., over-attentiveness to irrelevant language tokens or under-attentiveness to discriminative visual features) that drive object hallucination. To mitigate object hallucination, we further propose Attention Imbalance Rectification (AIR), a lightweight decoding-time intervention method that reallocates attention weights and adjusts attention distributions to rectify modality-wise and token-wise imbalances. Extensive evaluations on four mainstream LVLMs and three benchmarks (CHAIR, POPE, and MM-Vet) with seven baselines demonstrate that AIR consistently reduces object hallucination rates, achieving up to a 35.1% reduction compared to the baselines, while improving up to 15.9% of LVLMs' general capability across diverse vision-language tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
