arXiv 2605.309112026-05-29cs.CVcs.AI

什么使LVLMs更少产生幻觉？揭示影响幻觉鲁棒性的架构因素

What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness

School of Computer Science, Engineering Research Center of Machine Learning and Industry Intelligence, Sichuan University（计算机科学学院，机器学习与产业智能工程研究中心，四川大学）；School of Computer and Information Engineering, Xiamen University of Technology（计算机与信息工程学院，厦门理工大学）；Department of Electrical and Computer Engineering, University of Hong Kong（电气与计算机工程系，香港大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）

Yusheng He, Jizhe Zhou, Xia Du, Zheng Lin, Jun Luo, Jiancheng Lv

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文通过将架构设计分解为语言基础、视觉表示和语义对齐三个维度，并引入CoSimUE基准，系统探索了架构因素对LVLMs幻觉鲁棒性的影响，发现模型参数扩展效果有限，而增强视觉编码器、语言基础和语义对齐能分别减少不同类型的幻觉。

摘要：

幻觉仍然是削弱大型视觉-语言模型（LVLMs）可靠性的关键挑战之一。但什么使LVLM更少产生幻觉？许多现有工作专注于改进模型的内部组件。我们认为幻觉从根本上源于模型架构的设计方式。为了研究这一点，我们将架构设计分解为三个维度：语言基础（LF）、视觉表示（VR）和语义对齐（SA），并将幻觉分为共现型、相似型和先前被忽视的不确定型。基于这一框架，我们提出了CoSimUE基准，通过受控文本扰动和随机扰动创建细粒度的幻觉场景，从而建立设计选择与幻觉行为之间的映射。在7个设计方面的实验表明：1）广泛强调的参数规模扩展对减少所有三类幻觉的影响有限；2）更大且训练更好的语言基础可以减少共现型幻觉；3）更强的视觉编码器和更高的分辨率减轻相似型错误；4）有效的对齐策略缓解不确定型幻觉。5）此外，跨维度分析显示，联合增强视觉保真度和对齐质量能带来最全面的改进。本研究首次系统性地将架构级设计与幻觉鲁棒性联系起来，为开发可靠且高效的LVLMs提供了实用指导。

英文摘要：

Hallucination remains one of the key challenges undermining the reliability of Large Vision-Language Models (LVLMs). But what makes an LVLM hallucinate less? Many existing efforts focus on improving internal components of the model. We argue that hallucination fundamentally stems from how the model architecture is designed. To investigate this, we factor the architecture design into three dimensions: Linguistic Foundation (LF), Visual Representation (VR), and Semantic Alignment (SA), and categorize hallucinations into Co-occurrence, Similarity, and previously overlooked Uncertainty types. Building on this formulation, we propose CoSimUE, a benchmark that creates fine-grained hallucination scenarios through controlled textual perturbations and random perturbations, enabling mapping between design choices and hallucination behaviors. Experiments across 7 design aspects show that: 1) the widely emphasized scaling of model parameters has only limited impact on reducing all three types of hallucinations; 2) larger and better-trained language foundations can reduce co-occurrence hallucinations; 3) stronger visual encoders and higher resolutions mitigate similarity errors; 4) effective alignment strategies alleviate uncertainty hallucinations. 5) Furthermore, cross-dimensional analysis reveals that jointly enhancing visual fidelity and alignment quality yields the most comprehensive improvements. This study provides the first systematic exploration linking architecture-level design to hallucination robustness, offering practical guidance for developing reliable and efficient LVLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。