Recently, Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in multi-modal context comprehension. However, they still suffer from hallucination problems referring to generating inconsistent outputs with the image content. To mitigate hallucinations, previous studies mainly focus on retraining LVLMs with custom datasets. Although effective, they inherently come with additional computational costs. In this paper, we propose a training-free framework, \textbf{MVP}, that aims to reduce hallucinations by making the most of the innate capabilities of the LVLMs via \textbf{M}ulti-\textbf{V}iew Multi-\textbf{P}ath Reasoning. Specifically, we first devise a multi-view information-seeking strategy to thoroughly perceive the comprehensive information in the image, which enriches the general global information captured by the original vision encoder in LVLMs. Furthermore, during the answer decoding, we observe that the occurrence of hallucinations has a strong correlation with the certainty of the answer tokens. Thus, we propose multi-path reasoning for each information view to quantify and aggregate the certainty scores for each potential answer among multiple decoding paths and finally decide the output answer. By fully grasping the information in the image and carefully considering the certainty of the potential answers when decoding, our MVP can effectively reduce hallucinations in LVLMs.The extensive experiments verify that our proposed MVP significantly mitigates the hallucination problem across four well-known LVLMs. The source code is available at: \url{https://github.com/GasolSun36/MVP}.
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
The Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology(香港科技大学);Tencent AI Lab(腾讯AI实验室);Huazhong University of Science and Technology(华中科技大学);Tsinghua University(清华大学)
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
Southwestern University of Finance and Economics(西南财经大学);Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳高等研究院);Center for Future Media, University of Electronic Science and Technology of China(电子科技大学未来媒体中心);Tongji University(同济大学)
TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University(中山大学);The University of Hong Kong(香港大学);Meituan(美团);Inspur Database Technology(Inspur数据库技术);Guilin University of Electronic Technology(桂林电子科技大学);Shenzhen Loop Area Institute(深圳河套学院);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室)
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Nepal Applied Mathematics and Informatics Institute for Research(尼泊尔应用数学与信息技术研究所);GastroIntestinal Department, Dhulikhel Hospital(杜尔基hel医院消化内科);Univesity of Lausanne(洛桑大学);University of West Virginia(西弗吉尼亚大学);University of Utah(犹他大学);University of Aberdeen(阿伯丁大学)
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Tsinghua University(清华大学);Zhipu AI(智谱AI);Chongqing University(重庆大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
MALM: A Multi-Information Adapter for Large Language Models to Mitigate Hallucination
Beijing Institute of Technology(北京理工大学);The Open University(开放大学);Beijing Information Science and Technology University(北京信息科技大学);The Hong Kong Polytechnic University(香港理工大学)
Do I Really Know? Learning Factual Self-Verification for Hallucination Reduction
Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University (HBKU), Doha, Qatar. \,The authors contributed equally. \,The authors jointly supervised the work
Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院,沈阳,中国);National Frontiers Science Center for Industrial Intelligence and Systems optimization, Northeastern University, Shenyang, China(东北大学工业智能与系统优化国家级前沿科学中心,沈阳,中国)
Copy and Paste GAN: Face Hallucination from Shaded Thumbnails
Southeast University(东南大学);University of Technology Sydney(悉尼科技大学);Huazhong University of Science and Technology(华中科技大学);Nanjing University of Posts and Telecommunications(南京邮电大学);Australian National University(澳大利亚国立大学)
Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery
Tufts University(塔夫茨大学);KAUST(阿卜杜拉国王科技大学);Network Rail(英国铁路网公司);Zuse Institute Berlin(柏林祖泽研究所);NVIDIA Research(英伟达研究院);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Pasteur Labs(巴斯德实验室);Harvard University(哈佛大学);University of Southampton(南安普顿大学);Stanford University(斯坦福大学);University of Chicago(芝加哥大学);Santa Fe Institute(圣塔菲研究所);The University of Edinburgh(爱丁堡大学);Jožef Stefan Institute(约热夫·斯特凡研究所);Karolinska Institutet(卡罗林斯卡学院);The Alan Turing Institute(艾伦·图灵研究所);Oxford Immune Algorithmics(牛津免疫算法公司);Francis Crick Institute(弗朗西斯·克里克研究所)