arXiv 2607.041632026-07-07cs.CVcs.AI

SeeMe：通过有效的视觉令牌工程减轻大型视觉语言模型中的幻觉

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理国家重点实验室）；University of Pennsylvania（宾夕法尼亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；De Artificial Intelligence Lab（德人工智能实验室）

Kai Tang, Jinhao You, Bohua Zhang, Yichen Guo, Yiding Sun, Dongxu Zhang, Chenxi Li, Xiande Huang, Shanghang Zhang

备注：: 12 pages, 4 figures, 6 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究大型视觉语言模型易产生幻觉问题，提出无训练框架SeeMe，引入传统机器学习特征工程概念，经三阶段令牌工程重组视觉令牌抑制幻觉源，实验证明其能减轻幻觉并提高输出一致性。

摘要：

大型视觉语言模型在视觉理解任务中取得进展，但仍易产生幻觉。现有方法多在解码阶段干预，忽视了误导解码过程的无关或嘈杂视觉令牌这一幻觉源。为此提出SeeMe，将传统机器学习特征工程概念引入大型视觉语言模型，通过三阶段令牌工程重组视觉令牌以抑制幻觉源，实验表明该方法能持续减少幻觉并提高输出一致性。

英文摘要：

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。