当观察不足时：视觉注意结构揭示大语言模型中的幻觉
When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
AI总结：
本文通过分析视觉注意的高频结构揭示大语言模型中的幻觉现象，提出LaSCD解码策略，有效减少幻觉并保持模型能力。
摘要：
多模态大语言模型（MLLMs）已成为视觉推理和基于问题的回答的关键接口，但它们仍易受视觉幻觉影响，即生成的回答与图像内容矛盾或提及不存在的对象。核心挑战在于幻觉并非总是由缺乏视觉注意引起：模型可能仍对图像标记分配大量注意质量，但内部却倾向于错误答案。本文表明，通过层间拉普拉斯能量测量视觉注意的高频结构，可以揭示幻觉发生层和真实答案短暂恢复层。基于此发现，我们提出LaSCD（拉普拉斯-谱对比解码），一种无需训练的解码策略，通过拉普拉斯能量选择信息层并闭合形式重映射下一个token的logits。在幻觉和通用多模态基准测试中，LaSCD一致减少幻觉同时保持通用能力，突显其作为忠实解码范式的潜力。代码可在https://github.com/macovaseas/LaSCD获取。
英文摘要：
Multimodal large language models (MLLMs) have become a key interface for visual reasoning and grounded question answering, yet they remain vulnerable to visual hallucinations, where generated responses contradict image content or mention nonexistent objects. A central challenge is that hallucination is not always caused by a simple lack of visual attention: the model may still assign substantial attention mass to image tokens while internally drifting toward an incorrect answer. In this paper, we show that the high-frequency structure of visual attention, measured by layer-wise Laplacian energy, reveals both the layer where hallucinated preferences emerge and the layer where the ground-truth answer transiently recovers. Building on this finding, we propose LaSCD (Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via Laplacian energy and remaps next-token logits in closed form. Experiments on hallucination and general multimodal benchmarks show that LaSCD consistently reduces hallucination while preserving general capabilities, highlighting its potential as a faithful decoding paradigm. The code is available at https://github.com/macovaseas/LaSCD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
