自校正解码与生成反馈：缓解大型视觉-语言模型中的幻觉
Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
AI总结：
提出DeGF，一种利用文本到图像生成模型反馈的无训练解码算法，通过生成图像作为视觉参考并采用互补或对比解码，有效缓解LVLMs中的幻觉，在六个基准上超越现有方法。
摘要：
尽管近期的大型视觉-语言模型（LVLMs）在多模态任务中展现出卓越的性能，但它们容易生成与给定视觉输入不一致的幻觉性文本响应，这限制了它们在实际场景中的适用性。受文本到图像生成过程是LVLMs中图像条件响应生成的逆过程这一观察的启发，我们探索了利用文本到图像生成模型来帮助缓解LVLMs中幻觉的潜力。我们发现，生成模型能够在响应级别和令牌级别提供有价值的自我反馈，以缓解幻觉。基于这一洞察，我们引入了自校正解码与生成反馈（DeGF），一种新颖的无训练算法，它将文本到图像生成模型的反馈整合到解码过程中，以有效缓解LVLMs中的幻觉。具体来说，DeGF从LVLMs生成的初始响应中生成一张图像，作为辅助视觉参考，并通过互补或对比解码提供自我反馈来验证和校正初始响应。大量实验结果表明，我们的方法在缓解多种类型的幻觉方面有效，并在六个基准上持续超越最先进的方法。代码可在 https://github.com/zhangce01/DeGF 获取。
英文摘要：
While recent Large Vision-Language Models (LVLMs) have shown remarkable performance in multi-modal tasks, they are prone to generating hallucinatory text responses that do not align with the given visual input, which restricts their practical applicability in real-world scenarios. In this work, inspired by the observation that the text-to-image generation process is the inverse of image-conditioned response generation in LVLMs, we explore the potential of leveraging text-to-image generative models to assist in mitigating hallucinations in LVLMs. We discover that generative models can offer valuable self-feedback for mitigating hallucinations at both the response and token levels. Building on this insight, we introduce self-correcting Decoding with Generative Feedback (DeGF), a novel training-free algorithm that incorporates feedback from text-to-image generative models into the decoding process to effectively mitigate hallucinations in LVLMs. Specifically, DeGF generates an image from the initial response produced by LVLMs, which acts as an auxiliary visual reference and provides self-feedback to verify and correct the initial response through complementary or contrastive decoding. Extensive experimental results validate the effectiveness of our approach in mitigating diverse types of hallucinations, consistently surpassing state-of-the-art methods across six benchmarks. Code is available at https://github.com/zhangce01/DeGF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
