Large vision-language models (LVLMs) have made significant progress in recent years. While LVLMs exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, they are prone to producing hallucinations. While several methods are proposed to evaluate the hallucinations in LVLMs, most are reference-based and depend on external tools, which complicates their practical application. To assess the viability of alternative methods, it is critical to understand whether the reference-free approaches, which do not rely on any external tools, can efficiently detect hallucinations. Therefore, we initiate an exploratory study to demonstrate the effectiveness of different reference-free solutions in detecting hallucinations in LVLMs. In particular, we conduct an extensive study on three kinds of techniques: uncertainty-based, consistency-based, and supervised uncertainty quantification methods on four representative LVLMs across two different tasks. The empirical results show that the reference-free approaches are capable of effectively detecting non-factual responses in LVLMs, with the supervised uncertainty quantification method outperforming the others, achieving the best performance across different settings.
Evaluation and Analysis of Hallucination in Large Vision-Language Models
Beijing Jiaotong University(北京交通大学);Xi’an Jiaotong University(西安交通大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shandong University(山东大学)
MedHallTune: An Instruction-Tuning Benchmark for Mitigating Medical Hallucination in Vision-Language Models
The Chinese University of Hong Kong(香港中文大学);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与XR研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
Tsinghua University, Tencent(清华大学、腾讯);Tencent(腾讯);Tencent, University of Macau(腾讯、澳门大学);University of Science and Technology Beijing(北京科技大学);Tsinghua University(清华大学)
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Computing Science and Technology, Fudan University(复旦大学计算机科学技术学院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Computing Science and Technology, Zhejiang Gongshang University(浙江工商大学计算机科学与技术学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系)
Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
Ministry of Education Key Laboratory of Intelligent Networks and Network Security(教育部智能网络与网络安全重点实验室);Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research(前沿人工智能研究中心,高性能计算研究所,科技研究局)
Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models
University of Helsinki(赫尔辛基大学);Politecnico di Torino(都灵理工大学);Université Bretagne Sud(南布列塔尼大学);University of Copenhagen(哥本哈根大学);University Grenoble Alpes(格勒诺布尔阿尔卑斯大学);University of Lorraine(洛林大学)
Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow
Cyberspace Institute of Advanced Technology, Guangzhou University(广州大学先进网络空间研究院);Huangpu Research School of Guangzhou University(广州大学黄埔研究院);Beihang University(北京航空航天大学);University of the Chinese Academy of Sciences(中国科学院大学)
Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention
Xi’an Jiaotong University(西安交通大学);National Engineering Laboratory for Big Data Analytics(国家大数据分析工程实验室);Nanyang Technological University(南洋理工大学);Lenovo Research(联想研究院);University of Massachusetts Boston(马萨诸塞大学波士顿分校);Zhejiang University of Technology(浙江工业大学)
Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);New Laboratory of Pattern Recognition (NLPR)(模式识别国家重点实验室(NLPR))
Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning
Beijing Key Lab of Traffic Data Analysis and Mining(北京交通大数据分析与挖掘重点实验室);Beijing Jiaotong University(北京交通大学);School of Information Technology and Management(信息科学技术学院)
TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
Drexel University(德雷塞尔大学);University of Electronic Science and Technology of China(电子科技大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));LLNL(劳伦斯利弗莫尔国家实验室);Lehigh University(莱斯大学)
Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)(模式识别新技术实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Kuaishou Technology(快手科技);Nanjing University(南京大学)
本文提出 Mixture of Decoding(MoD),通过比较原始与被注意图像 token 的输出一致性,自适应采用互补或对比解码策略,从而在多个基准上有效缓解 LVLMs 幻觉。
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);National University of Singapore(新加坡国立大学);Southeast University(东南大学);Wuhan AI Research(武汉人工智能研究所)