Large vision-language models (LVLMs) have demonstrated remarkable multimodal comprehension and reasoning capabilities, but they still suffer from severe object hallucination. Previous studies primarily attribute the flaw to linguistic prior caused by the scale mismatch between visual encoders and large language models (LLMs) in LVLMs. Specifically, as current LVLMs are built upon LLMs, they tend to over-rely on textual prompts and internal knowledge of LLMs, generating descriptions inconsistent with visual cues. However, through an in-depth investigation of the hallucinated mechanisms, we empirically reveal a previously overlooked phenomenon: LVLMs may ignore not only visual information but also textual modality during hallucination, a behavior termed as modality bias, which indicates that LVLMs struggle to simultaneously attend to both visual and textual modalities, leading to fragmented understanding of user-provided instructions. Based on this observation, we propose a simple yet effective training-free method to mitigate object hallucination. Concretely, we intervene and adjust the attention weights of textual and visual tokens, balancing cross-modal compatibility for better alignment with user intentions. Furthermore, we adopt a contrastive decoding strategy to reduce the LVLM's overreliance on its parametric knowledge, synergistically enhancing our attention manipulation. Extensive experiments confirm the widespread presence of modality bias in LVLMs. Notably, our method effectively mitigates hallucination across multiple open-source LVLMs and benchmarks, highlighting its generalizability and efficacy.
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
School of Intelligent Science and Technology, Nanjing University（南京大学智能科学与技术学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学医疗数据科学国家研究院）；School of Biomedical Engineering, Nanjing University（南京大学生物医学工程学院）
Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
School of Future Technology, Shanghai University（上海大学未来技术学院）；Institute of Artificial Intelligence, Shanghai University（上海大学人工智能研究院）；School of Mechatronic Engineering and Automation（机电工程与自动化学院）
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University（计算机科学与技术系，人工智能研究院，清华大学）；Gaoling School of Artificial Intelligence, Renmin University of China（人工智能学院，中国人民大学）；Kuaishou Technology Inc.（快手科技有限公司）；Shanghai Key Laboratory of Multi. Info. Processing, East China Normal University（多信息处理重点实验室，华东师范大学）
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Tsinghua University（清华大学）；Tsinghua University, Tencent（清华大学腾讯）；Tencent（腾讯）；University of Science and Technology Beijing（北京科技大学）；University of Macau（澳门大学）
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；National University of Singapore（新加坡国立大学）；Southeast University（东南大学）；Wuhan AI Research（武汉人工智能研究所）
TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
Drexel University（德雷塞尔大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；LLNL（劳伦斯利弗莫尔国家实验室）；Lehigh University（莱斯大学）
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computing Science and Technology, Fudan University（复旦大学计算机科学技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Computing Science and Technology, Zhejiang Gongshang University（浙江工商大学计算机科学与技术学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）
Evaluation and Analysis of Hallucination in Large Vision-Language Models
Beijing Jiaotong University（北京交通大学）；Xi’an Jiaotong University（西安交通大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shandong University（山东大学）
