ASCD：用于减少多模态大语言模型（MLLM）幻觉的注意力可导向对比解码
ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM
AI总结：
该研究针对MLLM的幻觉问题，提出ASCD方法，通过正负引导调整解码时的注意力分数，在多基准上显著减少幻觉并提升VQA准确率，且无需额外训练。
摘要：
多模态大语言模型（MLLM）常因过度依赖虚假视觉线索而产生幻觉。现有解决方法如视觉对比解码（VCD）和指令对比解码（ICD）虽能缓解该问题，但机制尚不明确。本文首先通过实验表明，这些方法的改进与跨模态注意力的重新分配存在系统性关联。基于此，我们提出注意力可导向对比解码（ASCD），其在解码过程中直接引导注意力分数。ASCD结合两部分：一是正引导，即放大自动挖掘的、在模型内部稳定且跨领域鲁棒的以文本为中心的注意力头；二是负引导，即抑制实时识别的关键视觉 token。该方法仅产生可忽略的运行时间和内存开销，无需额外训练。在五个MLLM主干和三种解码方案下，ASCD在POPE、CHAIR和MMHal-Bench上最多减少38.2%的幻觉，同时提升MMMU、MM-VET、ScienceQA、TextVQA和GQA等标准VQA基准的准确率。这些结果表明，注意力引导是一种简单、模型无关且有原则的路径，可实现更安全、更忠实的多模态生成。
英文摘要：
Multimodal large language models (MLLMs) frequently hallucinate by over-committing to spurious visual cues. Prior remedies-Visual and Instruction Contrastive Decoding (VCD, ICD)-mitigate this issue, yet the mechanism remains opaque. We first empirically show that their improvements systematically coincide with redistributions of cross-modal attention. Building on this insight, we propose Attention-Steerable Contrastive Decoding (ASCD), which directly steers the attention scores during decoding. ASCD combines (i) positive steering, which amplifies automatically mined text-centric heads-stable within a model and robust across domains-with (ii) negative steering, which dampens on-the-fly identified critical visual tokens. The method incurs negligible runtime and memory overhead and requires no additional training. Across five MLLM backbones and three decoding schemes, ASCD reduces hallucination on POPE, CHAIR, and MMHal-Bench by up to 38.2 percent while improving accuracy on standard VQA benchmarks, including MMMU, MM-VET, ScienceQA, TextVQA, and GQA. These results position attention steering as a simple, model-agnostic, and principled route to safer, more faithful multimodal generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
