arXiv 2606.006222026-05-30cs.CV

MM-Snowball：多模态多轮对话中的幻觉雪崩评估与缓解

MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue

College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Southern University of Science and Technology（南方科技大学）；TMLR Group, Hong Kong Baptist University（香港 Baptist 大学 TMLR 团体）；MM Lab, CUHK（CUHK 多模态实验室）；RAMS Lab, Huawei Technologies Co., Ltd.（华为技术有限公司 RAMS 实验室）

Yue Jiang, Xue Jiang, Lihua Zhang, Zhiqiang Wang, Yuhang Lu, Peng Wang, Bo Han, Feng Zheng, Dingkang Yang

备注：: Accepted by The International Conference on Machine Learning (ICML 2026)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型在多轮对话中因初始错误累积导致幻觉雪崩的问题，提出首个细粒度诊断基准MM-Snowball，并设计无训练的冲突感知视觉校正方法CAVR，通过表示级刷新视觉锚定和logit级修正输出分布来缓解雪崩效应。

摘要：

多模态大语言模型（MLLMs）展现出显著的视觉理解能力，但在交互环境中的可靠性受到幻觉雪崩的严重破坏：一种初始错误在对话轮次间放大，导致连贯性崩溃的现象。这种失败揭示了一个根本性的脆弱性，即模型逐渐忽视视觉锚定，转而过度依赖受污染的文本历史。现有基准主要局限于单轮VQA，无法捕捉长程交互中错误传播的复杂动态。为解决这一问题，我们引入了MM-Snowball，这是首个用于细粒度诊断对话中幻觉雪崩的基准。广泛评估表明，我们的基准对即使是先进的MLLMs也构成了重大挑战，并揭示了现有为单轮VQA设计的缓解方法的无效性。为对抗这种退化，我们提出了冲突感知视觉校正（CAVR）。这种无训练方法通过协同双机制缓解雪崩：在表示级刷新视觉锚定，并在logit级修正输出分布，有效地将模型重新锚定到视觉事实。实验表明，CAVR达到了最先进的性能，为更可靠的交互式AI提供了一条有希望的路径。数据和代码可在 https://frenkie-chiang.github.io/MM-Snowball 获取。

英文摘要：

Multimodal large language models (MLLMs) demonstrate remarkable visual understanding, yet their reliability in interactive settings is severely undermined by hallucination snowballing: a phenomenon where initial errors amplify across conversational turns, leading to a collapse in coherence. This failure reveals a fundamental vulnerability where models progressively neglect visual grounding in favor of over-relying on polluted textual history. Existing benchmarks are predominantly confined to single-turn VQA, which fail to capture the complex dynamics of error propagation in long-horizon interactions. To address this, we introduce MM-Snowball, the first benchmark for fine-grained diagnosis of hallucination snowballing within dialogues. Extensive evaluation shows that our benchmark poses a significant challenge even to advanced MLLMs and reveals the inefficacy of existing mitigation methods designed for single-turn VQA. To counteract this degradation, we propose Conflict-Aware Visual Rectification (CAVR). This training-free method mitigates snowballing through a synergistic dual-mechanism that refreshes visual grounding at the representation level and rectifies output distributions at the logit level, effectively re-anchoring the model to visual facts. Experiments demonstrate that CAVR achieves state-of-the-art performance, offering a promising path toward more reliable interactive AI. Data and code are available at: https://frenkie-chiang.github.io/MM-Snowball

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。