VaLiD：通过视觉层融合对比解码缓解大型视觉语言模型的幻觉
VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
AI总结：
针对大型视觉语言模型因视觉编码扭曲导致的幻觉问题，提出无需训练的 VaLiD，利用不确定性选择并融合视觉层进行对比解码，在多个基准上达到最先进性能。
摘要：
大型视觉语言模型（LVLMs）在多模态任务推理中展现出卓越能力。然而，它们经常生成看似合理却不能准确反映视觉内容的回答，这一现象被称为幻觉。近期方法引入了无需训练的途径，通过在推理阶段调整解码策略来缓解幻觉，并且通常将幻觉归因于语言模型本身。但我们的分析表明，视觉编码过程中的扭曲会显著影响模型的推理能力。具体而言，较早的视觉层可能保留关键特征，但随着信息向输出层传播，这些特征会逐渐扭曲。基于这些洞见，我们从视觉编码角度提出一种新颖的幻觉缓解方法：Visual Layer Fusion Contrastive Decoding（视觉层融合对比解码，VaLiD）。该方法利用不确定性指导视觉层选择，纠正视觉编码过程中的扭曲，从而提升生成内容的可靠性。实验结果表明，VaLiD 在多个基准上有效缓解幻觉，并且与基线方法相比取得了最先进性能。代码可在 Github 获取：https://github.com/RicardoLuL/VaLiD_LVLMs_hallucinations。
英文摘要：
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal task reasoning. However, they often generate responses that appear plausible yet do not accurately reflect the visual content, a phenomenon known as hallucination. Recent approaches have introduced training-free methods to mitigate hallucinations by adjusting the decoding strategy during the inference stage, typically attributing hallucinations to the language model itself. Our analysis, however, reveals that distortions in the visual encoding process significantly affect the model's reasoning capabilities. Specifically, earlier visual layers may retain key features but gradually distort as the information propagates toward the output layer. Building on these insights, we propose a novel hallucination-mitigation method from the visual encoding perspective: \textbf{V}isu\textbf{a}l \textbf{L}ayer Fus\textbf{i}on Contrastive \textbf{D}ecoding (\textbf{VaLiD}). This method utilizes uncertainty to guide the visual layer selection, correcting distortions in the visual encoding process and thereby enhancing the reliability of the generated content. Experimental results demonstrate the effectiveness of VaLiD in mitigating hallucinations across various benchmarks, achieving state-of-the-art performance when compared to baseline methods. Codes are available at \href{https://github.com/RicardoLuL/VaLiD_LVLMs_hallucinations}{Github}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
