arXiv 2604.097492026-04-10cs.CV

看清真相：公平关注提升 groundedness 并减少 hallucination 在视觉语言对齐中

See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学，阿布扎比，阿联酋）；King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia（法赫德国王石油矿产大学，达兰，沙特阿拉伯）；Macquarie University, Sydney, Australia（麦考瑞大学，悉尼，澳大利亚）；University of Surrey, Guildford, United Kingdom（萨里大学，吉尔福德，英国）；University of New South Wales, Sydney, Australia（新南威尔士大学，悉尼，澳大利亚）

Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui, Adinath Madhavrao Dukre, Rafiq Ali, Sara Atito, Yutong Xie, Imran Razzak

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DOP-OBC方法，通过公平分配注意力减少视觉语言对齐中的幻觉问题，提升生成的准确性与一致性。

摘要：

多模态大语言模型（MLLMs）经常产生不存在于视觉输入中的对象幻觉，通常是因为解码时的注意力过度集中在视觉主导或频繁出现的内容上。我们观察到这种注意力分配的不平等是对象幻觉的根本原因：当罕见、小或上下文边缘化的对象获得的注意力不足时，模型无法将其生成 grounded 在完整的视觉场景中。我们主张图像中每个对象，无论其大小、频率或视觉显著性，都应享有平等的表示机会。为此，我们提出DOP-OBC，一种训练免费且架构无关的解码策略，基于公平注意力原则。两种互补的对象感知信号协同工作：Dominant Object Penalty（DOP）通过软压制注意力集中在视觉主导区域的集中，以及Outlier Boost Coefficient（OBC）通过增强注意力向罕见但自信检测到的对象。这些信号作为每行logit调制注入到因果注意力掩码中，不需要权重更新并保持自回归解码属性。在图像和视频MLLMs上的广泛实验显示，在CHAIR和POPE基准上对象幻觉显著减少，同时在GPT-4o评估的captioning质量方面，正确性、一致性、细节、上下文和时间维度上都有改进。DOP-OBC证明了注意力分配的公平性不仅是设计原则，也是实现更忠实多模态生成的有效路径。

英文摘要：

Multimodal large language models (MLLMs) frequently hallucinate objects that are absent from the visual input, often because attention during decoding is disproportionately drawn to visually dominant or frequently occurring content. We observe that this inequity in attention allocation is a root cause of object hallucination: when rare, small, or contextually peripheral objects receive insufficient attention, the model fails to ground its generation in the full visual scene. We argue that every object in an image, regardless of its size, frequency or visual salience, deserves equal representational opportunity during decoding. To this end, we propose DOP-OBC, a training-free and architecture-agnostic decoding strategy built on the principle of equitable attention. Two complementary object-aware signals work in tandem: a Dominant Object Penalty (DOP) that softly suppresses attention over-concentration on visually dominant regions, and an Outlier Boost Coefficient (OBC) that amplifies attention toward rare yet confidently detected objects. These signals are injected as per-row logit modulations within the causal attention mask, requiring no weight updates and preserving autoregressive decoding properties. Extensive experiments across image and video MLLMs demonstrate consistent reductions in object hallucination on CHAIR and POPE benchmarks, alongside improvements in GPT-4o assessed captioning quality across correctness, consistency, detail, context and temporal dimensions. DOP-OBC establishes that fairness in attention allocation is not merely a design principle but a practical and effective path toward more faithful multimodal generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。