Though advanced in understanding visual information with human languages, Large Vision-Language Models (LVLMs) still suffer from multimodal hallucinations. A natural concern is that during multimodal interaction, the generated hallucinations could influence the LVLMs' subsequent generation. Thus, we raise a question: When presented with a query relevant to the previously generated hallucination, will LVLMs be misled and respond incorrectly, even though the ground visual information exists? To answer this, we propose a framework called MMHalSnowball to evaluate LVLMs' behaviors when encountering generated hallucinations, where LVLMs are required to answer specific visual questions within a curated hallucinatory conversation. Crucially, our experiment shows that the performance of open-source LVLMs drops by at least $31\%$, indicating that LVLMs are prone to accept the generated hallucinations and make false claims that they would not have supported without distractions. We term this phenomenon Multimodal Hallucination Snowballing. To mitigate this, we further propose a training-free method called Residual Visual Decoding, where we revise the output distribution of LVLMs with the one derived from the residual visual input, providing models with direct access to the visual information. Experiments show that our method can mitigate more than $24\%$ of the snowballed multimodal hallucination while maintaining capabilities.
MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Southern University of Science and Technology（南方科技大学）；TMLR Group, Hong Kong Baptist University（香港 Baptist 大学 TMLR 团体）；MM Lab, CUHK（CUHK 多模态实验室）；RAMS Lab, Huawei Technologies Co., Ltd.（华为技术有限公司 RAMS 实验室）
Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析实验室）；King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹大学）；University of Electronic Science and Technology of China（中国电子科学技术大学）；University of Copenhagen（哥本哈根大学）
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computing Science and Technology, Fudan University（复旦大学计算机科学技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Computing Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
The Chinese University of Hong Kong（香港中文大学）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与XR研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Tsinghua University（清华大学）；Tsinghua University, Tencent（清华大学腾讯）；Tencent（腾讯）；University of Science and Technology Beijing（北京科技大学）；University of Macau（澳门大学）
TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
Drexel University（德雷塞尔大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；LLNL（劳伦斯利弗莫尔国家实验室）；Lehigh University（莱斯大学）
Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning
Beijing Key Lab of Traffic Data Analysis and Mining（北京交通大数据分析与挖掘重点实验室）；Beijing Jiaotong University（北京交通大学）；School of Information Technology and Management（信息科学技术学院）
Mitigating Hallucination in Large Language Models: A Capability-Oriented Survey on RAG, Reasoning, and Agentic Systems
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Computing, Dublin City University（都柏林城市大学计算机学院）；School of Public Health, Wuhan University（武汉大学公共卫生学院）；Insight Centre for Data Analytics, University of Galway（Galway大学数据分析研究中心）；Insight Centre for Data Analytics, Dublin City University（都柏林城市大学数据分析研究中心）
