通过语言对比解码（LCD）缓解大型视觉-语言模型（LVLMs）中的幻觉
Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
AI总结：
针对LVLMs的对象幻觉问题，本文提出语言对比解码（LCD）算法，依据LLM分布置信度调整输出，无需重训即可显著降低幻觉并提升描述质量。
摘要：
大型视觉-语言模型（LVLMs）是大型语言模型（LLMs）的扩展，可促进图像和文本输入的处理，从而扩展AI能力。然而，由于依赖文本线索以及习得的对象共现偏差，LVLMs受到对象幻觉问题的困扰。尽管大多数研究都在量化这些幻觉，但缓解策略仍然匮乏。本研究提出了一种语言对比解码（LCD）算法，该算法根据LLM分布置信度调整LVLM输出，有效减少对象幻觉。我们在主流LVLMs上展示了LCD的优势：在COCO验证集上，POPE F1分数最高提升4%，CHAIR分数最高降低36%，同时图像描述质量分数也有所提高。该方法无需复杂后处理或重新训练即可有效改进LVLMs，且易于应用于不同模型。研究结果凸显了进一步探索LVLM专用解码算法的潜力。
英文摘要：
Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
