arXiv 2410.067952025-08-12cs.CLcs.CV

从像素到词元：重新审视大型视觉语言模型中的对象幻觉

From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models

University of Chinese Academy of Sciences（中国科学院大学）；Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University（计算机科学与技术系，人工智能研究院，清华大学）；Gaoling School of Artificial Intelligence, Renmin University of China（人工智能学院，中国人民大学）；Kuaishou Technology Inc.（快手科技有限公司）；Shanghai Key Laboratory of Multi. Info. Processing, East China Normal University（多信息处理重点实验室，华东师范大学）

Yuying Shang, Xinyi Zeng, Yutao Zhu, Xiao Yang, Zhengwei Fang, Jingyuan Zhang, Jiawei Chen, Zinan Liu, Yu Tian

DOI：: 10.1145/3746027.3755728

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文从架构角度重新审视大型视觉语言模型中的幻觉问题，发现其主因在于视觉特征提取或解耦不足，并提出即插即用的微调策略PATCH，利用自适应虚拟词元从边界框提取对象特征以缓解幻觉，在多模态数据集上达到SOTA性能。

摘要：

大型视觉语言模型（LVLMs）中的幻觉是一个重大挑战，即生成视觉输入中未呈现的对象，这损害了其可靠性。近期研究常将幻觉归因于对视觉输入缺乏理解，却忽略了一个更根本的问题：模型无法有效提取或解耦视觉特征。本文从架构角度重新审视LVLMs中的幻觉，探究主要原因在于视觉编码器（特征提取）还是模态对齐模块（特征解耦）。基于初步调查的发现，我们提出了一种新颖的微调策略PATCH，以缓解LVLMs中的幻觉。这种即插即用的方法可集成到各种LVLMs中，利用自适应虚拟词元从边界框中提取对象特征，从而解决因视觉特征解耦不足导致的幻觉。PATCH在多个多模态幻觉数据集上取得了最先进的性能。我们希望该方法能为研究人员提供对LVLMs幻觉潜在原因的更深刻见解，促进该领域的进一步发展与创新。

英文摘要：

Hallucinations in large vision-language models (LVLMs) are a significant challenge, i.e., generating objects that are not presented in the visual input, which impairs their reliability. Recent studies often attribute hallucinations to a lack of understanding of visual input, yet ignore a more fundamental issue: the model's inability to effectively extract or decouple visual features. In this paper, we revisit the hallucinations in LVLMs from an architectural perspective, investigating whether the primary cause lies in the visual encoder (feature extraction) or the modal alignment module (feature decoupling). Motivated by our findings on the preliminary investigation, we propose a novel tuning strategy, PATCH, to mitigate hallucinations in LVLMs. This plug-and-play method can be integrated into various LVLMs, utilizing adaptive virtual tokens to extract object features from bounding boxes, thereby addressing hallucinations caused by insufficient decoupling of visual features. PATCH achieves state-of-the-art performance on multiple multi-modal hallucination datasets. We hope this approach provides researchers with deeper insights into the underlying causes of hallucinations in LVLMs, fostering further advancements and innovation in this field.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。