分析与缓解大型视觉语言模型中的对象幻觉
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
AI总结：
提出LURE算法，基于共现、不确定性和对象位置统计分析，事后修正LVLM对象幻觉，在六个开源模型上提升23%并排名第一。
摘要：
大型视觉语言模型（LVLMs）在理解视觉信息与人类语言方面展现出了卓越的能力。然而，LVLMs仍然存在对象幻觉问题，即生成包含图像中实际不存在对象的描述。这可能对许多视觉语言任务产生负面影响，如视觉摘要和推理。为解决此问题，我们提出了一种简单而强大的算法——LVLM幻觉修正器（LURE），通过重建较少幻觉的描述来事后纠正LVLMs中的对象幻觉。LURE基于对对象幻觉背后关键因素的严格统计分析，包括共现（某些对象在图像中频繁与其他对象一起出现）、不确定性（LVLM解码过程中具有较高不确定性的对象）以及对象位置（幻觉通常出现在生成文本的后半部分）。LURE还可以无缝集成到任何LVLMs中。我们在六个开源LVLMs上评估了LURE，在一般对象幻觉评估指标上比之前的最佳方法提高了23%。在GPT和人类评估中，LURE始终排名第一。我们的数据和代码可在https://github.com/YiyangZhou/LURE获取。
英文摘要：
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages. However, LVLMs still suffer from object hallucination, which is the problem of generating descriptions that include objects that do not actually exist in the images. This can negatively impact many vision-language tasks, such as visual summarization and reasoning. To address this issue, we propose a simple yet powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify object hallucination in LVLMs by reconstructing less hallucinatory descriptions. LURE is grounded in a rigorous statistical analysis of the key factors underlying object hallucination, including co-occurrence (the frequent appearance of certain objects alongside others in images), uncertainty (objects with higher uncertainty during LVLM decoding), and object position (hallucination often appears in the later part of the generated text). LURE can also be seamlessly integrated with any LVLMs. We evaluate LURE on six open-source LVLMs, achieving a 23% improvement in general object hallucination evaluation metrics over the previous best approach. In both GPT and human evaluations, LURE consistently ranks at the top. Our data and code are available at https://github.com/YiyangZhou/LURE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
