arXiv 2606.310542026-07-01cs.CVcs.AIcs.CLcs.MM

ADAPT: 通过偏好调优实现注意力动态对齐以增强多模态大模型的忠实性

ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

University of Science and Technology of China（中国科学技术大学）；Huawei Technologies Ltd.（华为技术有限公司）；State Key Laboratory of Communication Content Cognition, People’s Daily Online（人民网传播内容认知国家重点实验室）

Zhiyuan Yao, Zheren Fu, Zhixiao Zheng, Jiajun Li, Yi Tu, Zhendong Mao

备注：: Accepted by ECCV 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型生成中的幻觉问题，提出ADAPT框架，通过跨注意力视觉锚点、注意力监督推理和视觉注意力引导DPO直接干预文本到图像的跨注意力动态，在多个基准上将幻觉率降低40%-60%。

摘要：

多模态大语言模型（MLLMs）因产生与所提供图像不一致的内容而严重受幻觉问题困扰。在本文中，我们识别出幻觉的内部标志：生成过程中文本到图像交叉注意力的逐步退化，导致诸如注意力不集中或偏斜等特定失败模式。现有的缓解策略大多以结果为导向，并未明确针对这种失败模式。为解决此问题，我们提出ADAPT（注意力动态对齐与偏好调优），一个直接干预文本到图像交叉注意力动态的基于注意力的框架。我们提出的ADAPT包含三个关键贡献：从早期解码中提炼的交叉注意力视觉锚点，以提供稳定的空间基础；一种在线检测和纠正注意力漂移的注意力监督推理机制；以及一种视觉注意力引导DPO，将偏好与视觉基础响应对齐。实验表明，ADAPT的每个组件都有助于减少幻觉，完整框架在多个幻觉基准上取得了新的最佳结果，在主流骨干网络上将幻觉率降低40%-60%，同时保持通用的多模态能力。我们的工作通过探索模型内部的文本到图像交叉注意力行为，为缓解幻觉提供了基于注意力的视角。代码见此链接。

英文摘要：

Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention during generation, leading to specific failure patterns like unfocused or biased attention. Existing mitigation strategies are largely outcome-driven and do not explicitly target this failure mode. To address this problem, we propose ADAPT (Attention Dynamics Alignment with Preference Tuning), an attention-based framework that intervenes directly on text-to-image cross-attention dynamics. We propose ADAPT with three key contributions: a cross-attention visual anchor refined from early decoding to provide stable spatial grounding, an attention-supervised inference mechanism that detects and corrects attention drift online, and a Visual Attention Guidance DPO that aligns preferences toward visually grounded responses. Experiments show that each component of ADAPT contributes to hallucination reduction, and the full framework achieves new best results across multiple hallucination benchmarks, reducing hallucination rates by 40%-60% across mainstream backbones while preserving general multimodal capabilities. Our work provides an attention-based perspective on mitigating hallucinations by exploring the model's internal text-to-image cross-attention behaviors. Code is available at https://github.com/yao-ustc/ADAPT

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。