T. Nguyen, H. S. Yoon, T. M. Luu, E. Yoon and C. D. Yoo, "Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning," in IEEE Access, vol. 14, pp. 125177-125195, 2026
Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41\% on the COCO dataset and 29\% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Tsinghua University(清华大学);Tsinghua University, Tencent(清华大学腾讯);Tencent(腾讯);University of Science and Technology Beijing(北京科技大学);University of Macau(澳门大学)
Evaluation and Analysis of Hallucination in Large Vision-Language Models
Beijing Jiaotong University(北京交通大学);Xi’an Jiaotong University(西安交通大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shandong University(山东大学)
Visual hallucination detection in large vision-language models via evidential conflict
Beijing Key Lab of Traffic Data Mining(北京交通数据挖掘与具身智能重点实验室);State Key Laboratory of Advanced Rail Autonomous Operation(先进轨道交通自主运行国家重点实验室);School of Computer Science and Technology(计算机科学与技术学院);Beijing Jiaotong University(北京交通大学);School of Automation and Intelligence(自动化与智能学院);School of Electronic and Information Engineering(电子与信息工程学院)
SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
Department of Electrical and Computer Engineering, Northeastern University(电气与计算机工程系,东北大学);Khoury College of Computer Science, Northeastern University(计算机科学学院,东北大学)
Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation
Center for Future Media, University of Electronic Science and Technology of China(电子科技大学未来媒体中心);Southwestern University of Finance and Economics(西南财经大学);Tongji University(同济大学)
HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing
School of Information and Communication Technology, Griffith University, Australia(信息与通信技术学院,格里菲斯大学);Data61, CSIRO, Australia(Data61,澳大利亚联邦科学与工业研究组织);School of Engineering, Macquarie University, Sydney, Australia(工程学院,麦觉大学);School of Information Systems, Queensland University of Technology, Australia(信息系统学院,昆士兰技术大学)
Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University)(东南大学新一代人工智能技术及其交叉应用关键实验室);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Wuhan University of Technology(武汉理工大学);National University of Singapore(新加坡国立大学)
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
School of Intelligent Science and Technology, Nanjing University(南京大学智能科学与技术学院);National Institute of Healthcare Data Science at Nanjing University(南京大学医疗数据科学国家研究院);School of Biomedical Engineering, Nanjing University(南京大学生物医学工程学院)
CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs
Tsinghua University(清华大学);Pengcheng Laboratory(鹏城实验室);Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加州大学伯克利分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校);Chang’an University(长安大学)
Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
Shanghai Jiao Tong University(上海交通大学);Peking University(北京大学);Beijing University of Chemical Technology(北京化工大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学)
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University(中山大学);The University of Hong Kong(香港大学);Meituan(美团);Inspur Database Technology(Inspur数据库技术);Guilin University of Electronic Technology(桂林电子科技大学);Shenzhen Loop Area Institute(深圳河套学院);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室)
Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);New Laboratory of Pattern Recognition (NLPR)(模式识别国家重点实验室(NLPR))
Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs
School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);College of Computer and Information, Hohai University(河海大学计算机与信息学院);Institute for Infocomm Research, A*STAR(A*STAR信息与通信研究所)
Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steering
School of Electronic Engineering, Xidian University, Xi'an, China(西安电子科技大学电子工程学院);School of Computer Science and Technology, Xidian University, Xi'an, China(西安电子科技大学计算机科学与技术学院);College of Computer and Information, Hohai University, Nanjing, China(河海大学计算机与信息学院);Institute for Infocomm Research, A*STAR, Singapore(新加坡资讯研究院)
Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Macquarie University(麦考瑞大学);York University(约克大学);Northern Illinois University(北伊利诺伊大学);University of Technology Sydney(悉尼科技大学);North South University(北南大学);Lancaster University(兰卡斯特大学);Australian National University(澳大利亚国立大学)