Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
The Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology(香港科技大学);Tencent AI Lab(腾讯AI实验室);Huazhong University of Science and Technology(华中科技大学);Tsinghua University(清华大学)
While Large Vision-Language Models (LVLMs) have rapidly advanced in recent years, the prevalent issue known as the `hallucination' problem has emerged as a significant bottleneck, hindering their real-world deployments. Existing methods mitigate this issue mainly from two perspectives: One approach leverages extra knowledge like robust instruction tuning LVLMs with curated datasets or employing auxiliary analysis networks, which inevitable incur additional costs. Another approach, known as contrastive decoding, induces hallucinations by manually disturbing the vision or instruction raw inputs and mitigates them by contrasting the outputs of the disturbed and original LVLMs. However, these approaches rely on empirical holistic input disturbances and double the inference cost. To avoid these issues, we propose a simple yet effective method named Self-Introspective Decoding (SID). Our empirical investigation reveals that pretrained LVLMs can introspectively assess the importance of vision tokens based on preceding vision and text (both instruction and generated) tokens. We develop the Context and Text-aware Token Selection (CT2S) strategy, which preserves only unimportant vision tokens after early layers of LVLMs to adaptively amplify text-informed hallucination during the auto-regressive decoding. This approach ensures that multimodal knowledge absorbed in the early layers induces multimodal contextual rather than aimless hallucinations. Subsequently, the original token logits subtract the amplified vision-and-text association hallucinations, guiding LVLMs decoding faithfully. Extensive experiments illustrate SID generates less-hallucination and higher-quality texts across various metrics, without extra knowledge and much additional computation burdens.
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
Southwestern University of Finance and Economics(西南财经大学);Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(电子科技大学深圳高等研究院);Center for Future Media, University of Electronic Science and Technology of China(电子科技大学未来媒体中心);Tongji University(同济大学)
TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室)
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
Sun Yat-sen University(中山大学);The University of Hong Kong(香港大学);Meituan(美团);Inspur Database Technology(Inspur数据库技术);Guilin University of Electronic Technology(桂林电子科技大学);Shenzhen Loop Area Institute(深圳河套学院);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室)
VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering
Department of Data Science \& AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia School of Translational Medicine, Faculty of Medicine, Nursing;Health Sciences, Monash University, Melbourne, VIC 3800, Australia Hong Kong Polytechnic University, Hong Kong SAR, China
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Nepal Applied Mathematics and Informatics Institute for Research(尼泊尔应用数学与信息技术研究所);GastroIntestinal Department, Dhulikhel Hospital(杜尔基hel医院消化内科);Univesity of Lausanne(洛桑大学);University of West Virginia(西弗吉尼亚大学);University of Utah(犹他大学);University of Aberdeen(阿伯丁大学)
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Tsinghua University(清华大学);Zhipu AI(智谱AI);Chongqing University(重庆大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
MALM: A Multi-Information Adapter for Large Language Models to Mitigate Hallucination
Beijing Institute of Technology(北京理工大学);The Open University(开放大学);Beijing Information Science and Technology University(北京信息科技大学);The Hong Kong Polytechnic University(香港理工大学)
Joint Knowledge Base Completion and Question Answering by Combining Large Language Models and Small Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院,沈阳,中国);National Frontiers Science Center for Industrial Intelligence and Systems optimization, Northeastern University, Shenyang, China(东北大学工业智能与系统优化国家级前沿科学中心,沈阳,中国)