arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.17150cs.CVcs.AI
收藏

观察、比较、决策:通过多视角多路径推理缓解大型视觉语言模型中的幻觉问题

Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning

Xiamen University(厦门大学);Huazhong University of Science and Technology(华中科技大学);The Chinese University of Hong Kong(香港中文大学)

Xiaoye Qu, Jiashuo Sun, Wei Wei, Yu Cheng

AI总结:

本文提出免训练框架MVP,通过多视角信息搜寻与多路径推理量化答案确定性,在不增加训练成本的情况下有效缓解大型视觉语言模型的幻觉问题。

摘要:

近年来,大型视觉语言模型(LVLMs)在多模态上下文理解方面展现出令人瞩目的能力。然而,它们仍然存在幻觉问题,即生成与图像内容不一致的输出。为了缓解幻觉,以往研究主要集中于使用定制数据集重新训练LVLMs。尽管这些方法有效,但它们本质上带来了额外的计算成本。在本文中,我们提出了一种免训练框架MVP,旨在通过多视角多路径推理(Multi-View Multi-Path Reasoning)充分利用LVLMs的固有能力来减少幻觉。具体而言,我们首先设计了一种多视角信息搜寻策略,以彻底感知图像中的全面信息,从而丰富LVLMs中原始视觉编码器所捕获的全局整体信息。此外,在答案解码过程中,我们观察到幻觉的发生与答案token的确定性具有很强的相关性。因此,我们针对每个信息视角提出多路径推理,以量化并聚合多个解码路径中每个潜在答案的确定性分数,并最终决定输出答案。通过在解码时充分掌握图像中的信息并仔细考虑潜在答案的确定性,我们的MVP能够有效减少LVLMs中的幻觉。大量实验验证了我们提出的MVP在四个知名LVLMs上显著缓解了幻觉问题。源代码可在以下地址获取:https://github.com/GasolSun36/MVP。

英文摘要:

Recently, Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in multi-modal context comprehension. However, they still suffer from hallucination problems referring to generating inconsistent outputs with the image content. To mitigate hallucinations, previous studies mainly focus on retraining LVLMs with custom datasets. Although effective, they inherently come with additional computational costs. In this paper, we propose a training-free framework, \textbf{MVP}, that aims to reduce hallucinations by making the most of the innate capabilities of the LVLMs via \textbf{M}ulti-\textbf{V}iew Multi-\textbf{P}ath Reasoning. Specifically, we first devise a multi-view information-seeking strategy to thoroughly perceive the comprehensive information in the image, which enriches the general global information captured by the original vision encoder in LVLMs. Furthermore, during the answer decoding, we observe that the occurrence of hallucinations has a strong correlation with the certainty of the answer tokens. Thus, we propose multi-path reasoning for each information view to quantify and aggregate the certainty scores for each potential answer among multiple decoding paths and finally decide the output answer. By fully grasping the information in the image and carefully considering the certainty of the potential answers when decoding, our MVP can effectively reduce hallucinations in LVLMs.The extensive experiments verify that our proposed MVP significantly mitigates the hallucination problem across four well-known LVLMs. The source code is available at: \url{https://github.com/GasolSun36/MVP}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑