arXiv 2505.166522025-06-07cs.CVcs.LG

看得远且清晰：利用注意力因果解码缓解 MLLMs 中的幻觉

Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

Monash University（蒙纳士大学）；MBZUAI；XJTLU（西安交通大学）；Shanghai Jiaotong University（上海交通大学）；Fudan University（复旦大学）；University of Minnesota（明尼苏达大学）；Cornell University（康奈尔大学）

Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zelin Peng, Zhiwei Yang, Jionglong Su, Minquan Lin, Yifan Peng, Xuelian Cheng, Imran Razzak, Zongyuan Ge

备注：: Clarification note for the CVPR 2025 paper (FarSight). Prepared by a subset of the original authors; remaining co-authors are acknowledged in the text

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对 MLLMs 的初始与滚雪球式幻觉，提出即插即用的 FarSight 因果掩码解码策略，通过注意力寄存器和递减掩码率的位置感知编码减少离群 token 干扰，并在图像与视频基准上验证其有效性。

摘要：

多模态大语言模型（MLLMs）的最新进展显著提升了视觉问答任务的性能。然而，这些模型经常受到幻觉问题的困扰。在本研究中，幻觉被划分为两种主要类型：初始幻觉和滚雪球式幻觉。我们认为，可以直接从 token 交互过程中提取充分的上下文信息。受解码策略中因果推断的启发，我们提出利用因果掩码在多模态 token 之间建立信息传播。其假设是，这些 token 之间的交互不足可能导致模型依赖离群 token，而忽视密集且丰富的上下文线索。因此，我们建议通过处理离群 token 来干预传播过程，以增强上下文内推理。围绕这一目标，我们提出 FarSight，这是一种通用的即插即用解码策略，仅通过优化因果掩码即可减少来自离群 token 的注意力干扰。我们方法的核心是有效的 token 传播。我们在因果掩码的上三角矩阵中设计了一种注意力寄存器结构，动态分配注意力，以捕获转移到离群 token 的注意力。此外，我们还提出了一种具有递减掩码率的位置感知编码方法，使模型能够关注更远的前文 token，尤其适用于视频序列任务。通过大量实验，FarSight 在图像和视频基准测试上、跨不同 MLLMs 均展现出显著的幻觉缓解性能，证明了其有效性。

英文摘要：

Recent advancements in multimodal large language models (MLLMs) have significantly improved performance in visual question answering. However, they often suffer from hallucinations. In this work, hallucinations are categorized into two main types: initial hallucinations and snowball hallucinations. We argue that adequate contextual information can be extracted directly from the token interaction process. Inspired by causal inference in the decoding strategy, we propose to leverage causal masks to establish information propagation between multimodal tokens. The hypothesis is that insufficient interaction between those tokens may lead the model to rely on outlier tokens, overlooking dense and rich contextual cues. Therefore, we propose to intervene in the propagation process by tackling outlier tokens to enhance in-context inference. With this goal, we present FarSight, a versatile plug-and-play decoding strategy to reduce attention interference from outlier tokens merely by optimizing the causal mask. The heart of our method is effective token propagation. We design an attention register structure within the upper triangular matrix of the causal mask, dynamically allocating attention to capture attention diverted to outlier tokens. Moreover, a positional awareness encoding method with a diminishing masking rate is proposed, allowing the model to attend to further preceding tokens, especially for video sequence tasks. With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。