By leveraging both texts and images, large vision language models (LVLMs) have shown significant progress in various multi-modal tasks. Nevertheless, these models often suffer from hallucinations, e.g., they exhibit inconsistencies between the visual input and the textual output. To address this, we propose H-POPE, a coarse-to-fine-grained benchmark that systematically assesses hallucination in object existence and attributes. Our evaluation shows that models are prone to hallucinations on object existence, and even more so on fine-grained attributes. We further investigate whether these models rely on visual input to formulate the output texts.
Evaluating Object Hallucination in Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）；School of Information, Renmin University of China（中国人民大学信息学院）；Meituan Group（美团集团）
HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Jiaotong University（北京交通大学）
One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination
National Key Laboratory for Novel Software Technology, Nanjing University, China（南京大学新型软件技术国家重点实验室）；School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院）
SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理国家重点实验室）；University of Pennsylvania（宾夕法尼亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；De Artificial Intelligence Lab（德人工智能实验室）
Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
School of Future Technology, Shanghai University（上海大学未来技术学院）；Institute of Artificial Intelligence, Shanghai University（上海大学人工智能研究院）；School of Mechatronic Engineering and Automation（机电工程与自动化学院）
Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Models
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）；Alibaba Group（阿里巴巴集团）；Zhejiang University of Technology（浙江工业大学）；Binjiang Institute of Artificial Intelligence（滨江人工智能研究院）
SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucination
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Daegu Gyeongbuk Institute of Science and Technology（大邱庆州科学技术院）；IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University（IPAI、AIIS、ASRI、INMC 和 ISRC，首尔国立大学）
Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；School of Automation, Nanjing University of Information Science and Technology（南京信息工程大学自动化学院）；School of Automotive Studies, Tongji University（同济大学汽车学院）
POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation
Singapore University of Technology and Design（新加坡科技设计大学）；Tencent（腾讯）；Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；Lancaster University（兰卡斯特大学）
See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学，阿布扎比，阿联酋）；King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia（法赫德国王石油矿产大学，达兰，沙特阿拉伯）；Macquarie University, Sydney, Australia（麦考瑞大学，悉尼，澳大利亚）；University of Surrey, Guildford, United Kingdom（萨里大学，吉尔福德，英国）；University of New South Wales, Sydney, Australia（新南威尔士大学，悉尼，澳大利亚）
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
Peking University（北京大学）；University of Pennsylvania（宾夕法尼亚大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；Virginia Tech（弗吉尼亚理工大学）；University of Electronic Science and Technology of China（电子科技大学）；De Artificial Intelligence Lab（人工智能实验室）
