当RAG有害：诊断和缓解检索增强LVLMs中的注意力分散
When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
AI总结：
本文发现检索增强生成（RAG）在LVLMs中导致注意力分散（AD）问题，即检索文本抑制视觉注意力并偏离问题相关区域，提出MAD-RAG方法通过双问题公式和注意力混合来解耦视觉定位与上下文整合，在三个基准上提升性能并纠正大部分失败案例。
摘要：
虽然检索增强生成（RAG）是增强大型视觉语言模型（LVLMs）在基于知识的VQA任务上的主导范式之一，但最近的工作将RAG失败归因于对检索上下文的注意力不足，并提出减少分配给图像令牌的注意力。在这项工作中，我们识别了先前研究忽略的一个不同失败模式：注意力分散（AD）。当检索上下文足够（高度相关或包含正确答案）时，检索文本全局抑制视觉注意力，并且图像令牌上的注意力从问题相关区域转移。这导致模型在原本无需检索文本就能正确回答的问题上失败。为了缓解这个问题，我们提出了MAD-RAG，一种无需训练的干预方法，通过双问题公式解耦视觉定位与上下文整合，并结合注意力混合以保留图像条件证据。在OK-VQA、E-VQA和InfoSeek上的大量实验表明，MAD-RAG在不同模型家族中始终优于现有基线，相对于原始RAG基线分别取得了高达4.76%、9.20%和6.18%的绝对增益。值得注意的是，MAD-RAG纠正了高达74.68%的失败案例，且计算开销可忽略不计。
英文摘要：
While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens. In this work, we identify a distinct failure mode that previous study overlooked: Attention Distraction (AD). When the retrieved context is sufficient (highly relevant or including the correct answer), the retrieved text suppresses the visual attention globally, and the attention on image tokens shifts away from question-relevant regions. This leads to failures on questions the model could originally answer correctly without the retrieved text. To mitigate this issue, we propose MAD-RAG, a training-free intervention that decouples visual grounding from context integration through a dual-question formulation, combined with attention mixing to preserve image-conditioned evidence. Extensive experiments on OK-VQA, E-VQA, and InfoSeek demonstrate that MAD-RAG consistently outperforms existing baselines across different model families, yielding absolute gains of up to 4.76%, 9.20%, and 6.18% over the vanilla RAG baseline. Notably, MAD-RAG rectifies up to 74.68% of failure cases with negligible computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
