ConVis：基于幻觉可视化的对比解码以缓解多模态大语言模型中的幻觉
ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
AI总结：
为缓解多模态大语言模型幻觉问题，提出免训练对比解码方法ConVis，利用T2I模型重建图像并比较概率分布以惩罚幻觉生成，有效提升了模型可靠性。
摘要：
多模态大语言模型（MLLMs）中的幻觉现象（即生成的回答未能准确反映给定图像）对其可靠性构成了重大挑战。为了解决这一问题，我们提出了ConVis，一种新颖的免训练对比解码方法。ConVis利用文本到图像（T2I）生成模型，从幻觉描述中语义重建给定的图像。通过比较原始图像和重建图像产生的对比概率分布，ConVis使MLLMs能够捕获惩罚幻觉生成的视觉对比信号。值得注意的是，该方法完全在解码过程中运行，无需额外数据或模型更新。我们在五个流行基准上的大量实验表明，ConVis有效减少了各种MLLMs中的幻觉，突显了其提升模型可靠性的潜力。
英文摘要：
Hallucinations in Multimodal Large Language Models (MLLMs) where generated responses fail to accurately reflect the given image pose a significant challenge to their reliability. To address this, we introduce ConVis, a novel training-free contrastive decoding method. ConVis leverages a text-to-image (T2I) generation model to semantically reconstruct the given image from hallucinated captions. By comparing the contrasting probability distributions produced by the original and reconstructed images, ConVis enables MLLMs to capture visual contrastive signals that penalize hallucination generation. Notably, this method operates purely within the decoding process, eliminating the need for additional data or model updates. Our extensive experiments on five popular benchmarks demonstrate that ConVis effectively reduces hallucinations across various MLLMs, highlighting its potential to enhance model reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
