HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
Netaji Subhash Engineering College (NSEC)（奈尔贾伊·萨布哈工程学院）；TCG Crest；Electronics and Communication Sciences Unit (ECSU)（电子与通信科学单位）；Indian Statistical Institute（印度统计研究所）
Sujoy Nath, Arkaprabha Basu, Sharanya Dasgupta, Swagatam Das
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in vision-language understanding tasks. While these models often produce linguistically coherent output, they often suffer from hallucinations, generating descriptions that are factually inconsistent with the visual content, potentially leading to adverse consequences. Therefore, the assessment of hallucinations in MLLM has become increasingly crucial in the model development process. Contemporary methodologies predominantly depend on external LLM evaluators, which are themselves susceptible to hallucinations and may present challenges in terms of domain adaptation. In this study, we propose the hypothesis that hallucination manifests as measurable irregularities within the internal layer dynamics of MLLMs, not merely due to distributional shifts but also in the context of layer-wise analysis of specific assumptions. By incorporating such modifications, \textsc{\textsc{HalluShift++}} broadens the efficacy of hallucination detection from text-based large language models (LLMs) to encompass multimodal scenarios. Our codebase is available at https://github.com/C0mRD/HalluShift_Plus.
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（东华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of High Performance Computing, A*STAR（高性能计算研究所，A*STAR）
Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench
School of Biomedical Engineering, Southern Medical University（生物医学工程学院，南方医科大学）；School of Biomedical Engineering, Shanghai Jiaotong University（生物医学工程学院，上海交通大学）；Department of Electronic Engineering, Chinese University of Hong Kong（电子工程系，中国香港大学）；Faculty of Dentistry, The University of Hong Kong（牙科学院，香港大学）；Department of Nuclear Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（核医学科，广州中医药大学第二附属医院）；PET Center, Department of Nuclear Medicine, Guangdong Provincial People’s Hospital, Southern Medical University（PET中心，核医学科，广东省人民医院，南方医科大学）；Department of Nuclear Medicine, Nanfang Hospital, Southern Medical University（核医学科，南芳医院，南方医科大学）；Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospitals（核医学与分子影像学部，日内瓦大学医院）；Departments of Radiology, Physics, and Biomedical Engineering, The University of British Columbia（放射学、物理和生物医学工程系，不列颠哥伦比亚大学）；Medical Artificial Intelligence Laboratory, Westlake University（医学人工智能实验室，西湖大学）
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering
University of Science and Technology of China（中国科学技术大学）；Data Darkness Lab, MIRACLE Center, USTC（数据黑暗实验室，MIRACLE中心，中国科学技术大学）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Northwestern University（西北大学）；University of Alberta（阿尔伯塔大学）；Yale University（耶鲁大学）；Nanfang Hospital（南华医院）；Shenzhen University of Advanced Technology（深圳大学先进技术研究院）
WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发智能感知与认知联合实验室，中国科学技术大学）；ZheJiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）
