两种成因，而非一种：重新思考多模态大语言模型中的遗漏与编造幻觉
Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
AI总结：
该研究推翻MLLMs中遗漏与编造幻觉同源的假设，明确二者不同成因，提出视觉-语义注意力势场框架及即插即用的VPFC方法，可在不增加编造幻觉的前提下减少遗漏幻觉，为幻觉缓解研究指明新方向。
摘要：
多模态大语言模型（MLLMs）已取得令人瞩目的进展，但物体幻觉仍是一个长期存在的挑战。现有方法基于一个有缺陷的假设，即遗漏幻觉和编造幻觉具有共同成因，往往在减少遗漏的同时反而引发更多编造。在本研究中，我们推翻了这一观点，证明遗漏幻觉源于将感知到的视觉特征映射到语言表达时的置信度不足，而编造幻觉则是由训练语料中的统计偏差导致的跨模态表征空间内的虚假关联所引发。基于视觉注意力干预实验的发现，我们提出了视觉-语义注意力势场这一概念框架，它揭示了模型如何构建视觉证据以推断物体的存在与否。利用这一洞见，我们提出了视觉势场校准（VPFC）——一种即插即用的幻觉缓解方法，可有效减少遗漏幻觉，且不会引入额外的编造幻觉。我们的发现揭示了当前物体幻觉研究中的一个关键疏漏，并为开发更稳健、更均衡的幻觉缓解策略指明了新方向。
英文摘要：
Multimodal Large Language Models (MLLMs) have achieved impressive advances, yet object hallucination remains a persistent challenge. Existing methods, based on the flawed assumption that omission and fabrication hallucinations share a common cause, often reduce omissions only to trigger more fabrications. In this work, we overturn this view by demonstrating that omission hallucinations arise from insufficient confidence when mapping perceived visual features to linguistic expressions, whereas fabrication hallucinations result from spurious associations within the cross-modal representation space due to statistical biases in the training corpus. Building on findings from visual attention intervention experiments, we propose the Visual-Semantic Attention Potential Field, a conceptual framework that reveals how the model constructs visual evidence to infer the presence or absence of objects. Leveraging this insight, we introduce Visual Potential Field Calibration (VPFC), a plug-and-play hallucination mitigation method that effectively reduces omission hallucinations without introducing additional fabrication hallucinations. Our findings reveal a critical oversight in current object hallucination research and chart new directions for developing more robust and balanced hallucination mitigation strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
