基于内部事实的对比解码缓解大型视觉语言模型中的幻觉
Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
AI总结：
针对大型视觉语言模型的物体幻觉问题，提出无需训练的模型无关方法IFCD，通过对比扰动内部表示引发的幻觉分布来校准输出，有效移除幻觉对数几率，在POPE和MME基准上显著提升准确率。
摘要：
大型视觉语言模型（LVLMs）集成了视觉和语言模态，在各种多模态任务中展现出卓越的性能。然而，LVLMs 仍然容易受到物体幻觉问题的影响。以往缓解该问题的努力集中在监督微调（SFT）或引入外部知识上，这两者都会带来与训练和获取外部数据相关的巨大成本。为了应对这些挑战，我们提出了一种新颖的模型无关方法，称为 Internal Fact-based Contrastive Decoding（IFCD，基于内部事实的对比解码），旨在通过利用 LVLMs 自身的幻觉来缓解和抑制 LVLMs 推理过程中的幻觉。IFCD 基于实验观察，即对 LVLMs 内部表示的扰动往往会放大由语言偏差引起的幻觉。通过对比受扰动的分布，IFCD 校准 LVLMs 的输出，并有效地从最终预测中移除幻觉对数几率。实验结果验证了 IFCD 显著缓解了物体级别和属性级别的幻觉，并且与直接解码相比，在 POPE 上平均实现了 9% 的准确率提升，在 MME 物体幻觉子集上实现了 8% 的准确率提升。
英文摘要：
Large Visual Language Models (LVLMs) integrate visual and linguistic modalities, exhibiting exceptional performance across various multimodal tasks. Nevertheless, LVLMs remain vulnerable to the issue of object hallucinations. Previous efforts to mitigate this issue focus on supervised fine-tuning (SFT) or incorporating external knowledge, both of which entail significant costs related to training and the acquisition of external data. To address these challenges, we propose a novel model-agnostic approach termed Internal Fact-based Contrastive Decoding (IFCD), designed to mitigate and suppress hallucinations during the inference process of LVLMs by exploiting the LVLMs' own hallucinations. IFCD is grounded in experimental observations that alterations to the LVLMs' internal representations tend to amplify hallucinations caused by language bias. By contrasting disturbed distribution, IFCD calibrates the LVLMs' output and effectively removes the hallucinatory logits from the final predictions. Experimental results validate that IFCD significantly alleviates both object-level and attribute-level hallucinations while achieving an average 9% accuracy improvement on POPE and 8% accuracy improvement on MME object hallucinations subset compared with direct decoding, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
