NoLan: 通过动态抑制语言先验来缓解大视觉-语言模型中的对象幻觉
NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
AI总结：
NoLan通过动态抑制语言先验缓解大视觉-语言模型中的对象幻觉问题，有效提升模型准确性。
摘要：
对象幻觉是大视觉-语言模型（LVLMs）中的一个关键问题，其中输出包含在输入图像中不存在的对象。由此现象自然引发一个问题：LVLM管道中的哪个组件主要贡献于对象幻觉？是负责感知视觉信息的视觉编码器，还是负责生成文本响应的语言解码器？在本工作中，我们通过设计系统实验来分析视觉编码器和语言解码器在幻觉生成中的作用来回答这个问题。我们的观察表明，对象幻觉主要与语言解码器的强先验有关。基于这一发现，我们提出了一种简单且无需训练的框架，No-Language-Hallucination Decoding，NoLan，通过动态抑制语言先验来优化输出分布，该方法基于多模态输入与纯文本输入之间的输出分布差异进行调节。实验结果表明，NoLan在各种LVLMs上不同任务中有效减少了对象幻觉。例如，NoLan在POPE上取得了显著改进，使LLaVA-1.5 7B和Qwen-VL 7B的准确性分别提高了6.45和7.21。代码已公开在：https://github.com/lingfengren/NoLan。
英文摘要：
Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
