眼见为实：通过CLIP引导解码缓解大型视觉语言模型中的幻觉
Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
AI总结：
针对大型视觉语言模型的物体幻觉问题，提出免训练的CLIP引导解码（CGD）方法，利用CLIP相似度在解码时增强视觉基础，有效缓解幻觉并保持文本生成实用性。
摘要：
大型视觉语言模型（LVLMs）容易出现物体幻觉问题，即生成的文本包含不存在的物体，这极大限制了其可靠性与实用性。现有方法通常依赖模型的token似然值等内部信息、在额外数据集上进行指令微调，或引入复杂的外部工具。我们首先对句子级LVLM幻觉进行实证分析，发现与token似然值相比，图像的CLIP相似度是更强、更鲁棒的幻觉指标。受此启发，我们提出了CLIP引导解码（CGD）方法，这是一种简单但有效的免训练方法，用于在解码时减少物体幻觉。CGD利用CLIP通过增强生成文本与图像的视觉基础来引导模型的解码过程。实验表明，CGD能在多个LVLM家族中有效缓解物体幻觉，同时保持文本生成的实用性。代码可在https://github.com/d-ailin/CLIP-Guided-Decoding获取。
英文摘要：
Large Vision-Language Models (LVLMs) are susceptible to object hallucinations, an issue in which their generated text contains non-existent objects, greatly limiting their reliability and practicality. Current approaches often rely on the model's token likelihoods or other internal information, instruction tuning on additional datasets, or incorporating complex external tools. We first perform empirical analysis on sentence-level LVLM hallucination, finding that CLIP similarity to the image acts as a stronger and more robust indicator of hallucination compared to token likelihoods. Motivated by this, we introduce our CLIP-Guided Decoding (CGD) approach, a straightforward but effective training-free approach to reduce object hallucination at decoding time. CGD uses CLIP to guide the model's decoding process by enhancing visual grounding of generated text with the image. Experiments demonstrate that CGD effectively mitigates object hallucination across multiple LVLM families while preserving the utility of text generation. Codes are available at https://github.com/d-ailin/CLIP-Guided-Decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
