Large Multimodal Models (LMMs) often suffer from multimodal hallucinations, wherein they may create content that is not present in the visual inputs. In this paper, we explore a new angle of this issue: overly detailed training data hinders the model's ability to timely terminate generation, leading to continued outputs beyond visual perception limits. By investigating how the model decides to terminate generation with EOS, the special end-of-sentence token, we find that the model assesses the completeness of the entire sequence by comparing the generated text with the image. This observation suggests that the model possesses an inherent potential of making proper EOS decisions based on its visual perception to avoid overly lengthy outputs. To take advantage of such potential, we explore two methods to mitigate multimodal hallucinations: a training objective that enables the model to reduce hallucinations by learning from regular instruction data, and a data filtering strategy to prevent harmful training data from exacerbating model hallucinations. Both methods significantly improve the hallucination performance of LMMs, without requiring any additional data or knowledge.
Mitigating Object Hallucination via Robust Local Perception Search
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Center for Future Media & School of Computer Science and Engineering, University of Electronic Science and Technology of China（未来媒体中心及电子科技大学计算机科学与工程学院）
Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Models
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）；Alibaba Group（阿里巴巴集团）；Zhejiang University of Technology（浙江工业大学）；Binjiang Institute of Artificial Intelligence（滨江人工智能研究院）
Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
School of Future Technology, Shanghai University（上海大学未来技术学院）；Institute of Artificial Intelligence, Shanghai University（上海大学人工智能研究院）；School of Mechatronic Engineering and Automation（机电工程与自动化学院）
Mitigating Hallucinations in Large Vision-Language Models by Self-Injecting Hallucinations
Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information, CASIA（北京多模态信息超级智能安全重点实验室，中国科学院自动化所）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Hello Group（Hello集团）；Nanchang Hangkong University（南昌航空大学）；The University of Hong Kong（香港大学）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
School of Software, Beihang University（北京航空航天大学软件学院）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Tsinghua University（清华大学）；Zhipu AI（智谱AI）；Chongqing University（重庆大学）；Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能与数据分析实验室）；King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹大学）；University of Electronic Science and Technology of China（中国电子科学技术大学）；University of Copenhagen（哥本哈根大学）
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computing Science and Technology, Fudan University（复旦大学计算机科学技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Computing Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
The Chinese University of Hong Kong（香港中文大学）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与XR研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Mitigating Hallucination in Large Language Models: A Capability-Oriented Survey on RAG, Reasoning, and Agentic Systems
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Computing, Dublin City University（都柏林城市大学计算机学院）；School of Public Health, Wuhan University（武汉大学公共卫生学院）；Insight Centre for Data Analytics, University of Galway（Galway大学数据分析研究中心）；Insight Centre for Data Analytics, Dublin City University（都柏林城市大学数据分析研究中心）
TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
Drexel University（德雷塞尔大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；LLNL（劳伦斯利弗莫尔国家实验室）；Lehigh University（莱斯大学）
Exploring and Mitigating Fawning Hallucinations in Large Language Models
School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）；Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳MSU-BIT大学人工智能研究院）；Tianjin University（天津大学）；The Hong Kong University of Science（香港科学大学）
