RVSD：用于缓解大视觉语言模型中视觉幻觉的检索视觉稀疏解码

中文摘要 AI 辅助

大型视觉语言模型在视觉-语言任务中取得了显著成功，但仍易出现视觉幻觉（Visual Hallucinations, VHs），削弱了其在实际应用中的可靠性。现有解决方案通常需要精心整理的数据集、额外训练或多轮解码，导致可观的计算开销。本文提出RVSD（Retrieval Vision Sparse Decoding，检索视觉稀疏解码），这是一种无需训练、即插即用的解码框架，首次在单次解码过程中统一了 token 稀疏化与语义空间视觉检索（Semantic-Space Visual Retrieval, SSVR）。在RVSD中，我们引入语义导向的 token 选择策略，选择性地稀疏化冗余 token 同时保留关键视觉信息；进一步提出SSVR机制，将视觉补偿重新表述为共享语义空间内的按需跨模态检索过程。大量实验表明，RVSD在缓解视觉幻觉方面达到了最先进的性能，同时在长上下文生成设置下保持了稳健的抑制能力。

英文摘要

Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \textbf{RVSD} (\underline{R}etrieval \underline{V}ision \underline{S}parse \underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\footnote{https://github.com/canjie-liu/RVSD}

发表机构

• School of Automation, Guangdong University of Technology（广东工业大学自动化学院）
• City University of Hong Kong（香港城市大学）
• The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第二附属医院）

机构由 AI 辅助整理，请以论文原文为准。