通过图像锚定引导缓解大视觉语言模型中的物体幻觉
Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance
AI总结：
针对大视觉语言模型的物体幻觉问题，提出无需训练和API调用的MARINE框架，通过开源视觉模型提取物体级信息提供图像锚定引导，在5种主流LVLM上的表现优于现有微调方法，且能保持生成内容详细度。
摘要：
大视觉语言模型（LVLMs）的发展日益凸显出其倾向于在图像中幻觉出不存在物体的关键问题。为解决该问题，以往研究聚焦于使用专门策划的数据集或强大的大语言模型（LLM）来修正LVLM的输出。但这些方法要么需要高昂的训练或微调成本，要么需要调用专有LLM的API进行生成后校正。针对这些局限，我们提出了基于图像锚定引导的幻觉缓解框架MARINE（Mitigating hallucinAtion via image-gRounded guIdaNcE），该框架无需训练也无需API调用。MARINE通过在推理过程中为LVLM引入图像锚定引导，高效地减少物体幻觉，具体通过利用开源视觉模型提取物体级信息，从而提升LVLM生成内容的准确性。该框架的灵活性还支持集成多个视觉模型，以实现更可靠、鲁棒的物体级引导。我们在5种主流LVLM上，采用多种评估指标和基准开展了全面评估，结果证明了MARINE的有效性，其性能甚至优于现有的基于微调的方法。值得注意的是，在GPT-4V辅助的评估中，它在保持LVLM生成内容详细度的同时，持续减少了幻觉现象。我们已在https://github.com/Linxi-ZHAO/MARINE发布代码。
英文摘要：
The advancement of Large Vision-Language Models (LVLMs) has increasingly highlighted the critical issue of their tendency to hallucinate non-existing objects in the images. To address this issue, previous works focused on using specially curated datasets or powerful LLMs to rectify the outputs of LVLMs. However, these approaches require either costly training or fine-tuning, or API access to proprietary LLMs for post-generation correction. In response to these limitations, we propose Mitigating hallucinAtion via image-gRounded guIdaNcE (MARINE), a framework that is both training-free and API-free. MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs. This is achieved by leveraging open-source vision models to extract object-level information, thereby enhancing the precision of LVLM-generated content. Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance. Through comprehensive evaluations across 5 popular LVLMs with diverse evaluation metrics and benchmarks, we demonstrate the effectiveness of MARINE, which even outperforms existing fine-tuning-based methods. Remarkably, it reduces hallucinations consistently in GPT-4V-assisted evaluation while maintaining the detailedness of LVLMs' generations. We release our code at https://github.com/Linxi-ZHAO/MARINE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
