arXiv 2507.050562025-07-22cs.CVcs.AI

INTER：通过交互引导采样缓解大型视觉语言模型中的幻觉

INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling

University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）

Xin Dong, Shichao Dong, Jin Wang, Jing Huang, Li Zhou, Zenghui Sun, Lihua Jing, Jingsong Lan, Xiaoyong Zhu, Bo Zheng

备注：: Accepted by ICCV 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出免训练的交互引导采样算法INTER，通过引导LVLMs重新应用多模态交互信息来缓解幻觉，在六个基准上平均提升高达3.4%。

摘要：

大型视觉语言模型（LVLMs）中的幻觉给实际应用带来了重大挑战，因为LVLMs可能生成看似合理但与相关视觉内容不一致的回复。这一问题在人类认知中很少出现。我们认为，这种差异源于人类能够有效利用数据样本中的多模态交互信息。具体而言，人类通常首先收集多模态信息，分析跨模态之间的交互以进行理解，然后通过语言表达其理解。受这一观察启发，我们对流行的LVLMs进行了大量实验，获得的见解令人惊讶地揭示了LVLMs在多模态样本上具有类似人类但不太明显的认知行为。基于这些发现，我们进一步提出INTER：交互引导采样（Interaction Guidance Sampling），这是一种新颖的免训练算法，无需额外数据即可缓解幻觉。具体来说，INTER明确引导LVLMs在生成回复时有效地重新应用其对多模态交互信息的理解，从而减少潜在的幻觉。在包括VQA和图像描述任务在内的六个基准上，与最先进的解码策略相比，INTER在五个LVLMs上平均提升高达3.4%。代码将在论文被接收后发布。

英文摘要：

Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications, as LVLMs may generate responses that appear plausible yet remain inconsistent with the associated visual content. This issue rarely occurs in human cognition. We argue that this discrepancy arises from humans' ability to effectively leverage multimodal interaction information in data samples. Specifically, humans typically first gather multimodal information, analyze the interactions across modalities for understanding, and then express their understanding through language. Motivated by this observation, we conduct extensive experiments on popular LVLMs and obtained insights that surprisingly reveal human-like, though less pronounced, cognitive behavior of LVLMs on multimodal samples. Building on these findings, we further propose \textbf{INTER}: \textbf{Inter}action Guidance Sampling, a novel training-free algorithm that mitigate hallucinations without requiring additional data. Specifically, INTER explicitly guides LVLMs to effectively reapply their understanding of multimodal interaction information when generating responses, thereby reducing potential hallucinations. On six benchmarks including VQA and image captioning tasks, INTER achieves an average improvement of up to 3.4\% on five LVLMs compared to the state-of-the-art decoding strategy. The code will be released when the paper is accepted.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。