Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or "gaze shifts", during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead.
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学穆斯克特基金会数据科学研究所，香港大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程系，香港大学）
Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Computer Science and Engineering, University of Texas, Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Massachusetts General Hospital, Harvard Medical School（麻省总医院哈佛医学院）；Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）
Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm
Key Laboratory of Pervasive Computing, Tsinghua University（清华大学普适计算重点实验室）；The University of New South Wales（新南威尔士大学）；SKLSDE Lab, Beihang University（北航SKLSDE实验室）
SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
Department of Electrical and Computer Engineering, Seoul National University（首尔大学电气与计算机工程系）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能跨学科项目）
Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification
College of Computer Science, Sichuan University（四川大学计算机科学学院）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院）；Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University（四川大学数据保护与智能管理教育部重点实验室）；National Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology（北京理工大学自主智能无人系统国家重点实验室）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation
Department of Engineering Science, University of Oxford（工程科学系，牛津大学）；Department of Automation, Tsinghua University（自动化系，清华大学）；School of Information Science and Technology, Northwest University（信息科学与技术学院，西北大学）
UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Models
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（集成航空航天地面海洋大数据应用技术国家工程实验室，计算机科学与工程学院，西北工业大学）
Leveraging NTPs for Efficient Hallucination Detection in VLMs
Department of Data and Decision Science, Technion - Israel Institute of Technology（数据与决策科学系，技术学院-以色列理工学院）；Faculty of Computer and Information Science, Ben-Gurion University of the Negev（计算机与信息科学系，贝内-约尔根大学）
Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Models
Rochester Institute of Technology, Rochester, NY, USA（罗切斯特技术学院）；Nepal Applied Mathematics and Informatics Institute for Research (NAAMII)（尼泊尔应用数学与信息技术研究所）；Kathmandu University（加德满都大学）；University of Lausanne（洛桑大学）；University of Aberdeen（阿伯丁大学）
BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models
POSTECH（浦项科技大学）；Grad. School of AI, POSTECH（浦项科技大学人工智能学院）；Institute for Convergence Research and Education in Advanced Technology, Yonsei University（延世大学尖端技术融合研究教育学院）
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
Brown University（布朗大学）；Columbia University（哥伦比亚大学）；Emory University（埃默里大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；UC San Diego（圣地亚哥大学）
Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
AI4LIFE, Hanoi University of Science and Technology, Vietnam（AI4LIFE，越南科学与技术大学）；SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France（SAMOVAR，法国电信南巴黎学院，巴黎理工学院）；Military Central Hospital, Vietnam（越南108军中心医院）
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
School of Software, Beihang University（北京航空航天大学软件学院）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMs
Department of Computer Science and Technology, Kean University（计算机科学与技术系，凯恩大学）；Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，圣母大学）
Mitigating Image Captioning Hallucinations in Vision-Language Models
Department of Computer Science, The University of Alabama at Birmingham（亚拉巴马大学伯明翰分校计算机科学系）；Department of Computer Science, University of Waterloo（滑铁卢大学计算机科学系）
