We deal with the problem of information fusion driven satellite image/scene classification and propose a generic hallucination architecture considering that all the available sensor information are present during training while some of the image modalities may be absent while testing. It is well-known that different sensors are capable of capturing complementary information for a given geographical area and a classification module incorporating information from all the sources are expected to produce an improved performance as compared to considering only a subset of the modalities. However, the classical classifier systems inherently require all the features used to train the module to be present for the test instances as well, which may not always be possible for typical remote sensing applications (say, disaster management). As a remedy, we provide a robust solution in terms of a hallucination module that can approximate the missing modalities from the available ones during the decision-making stage. In order to ensure better knowledge transfer during modality hallucination, we explicitly incorporate concepts of knowledge distillation for the purpose of exploring the privileged (side) information in our framework and subsequently introduce an intuitive modular training approach. The proposed network is evaluated extensively on a large-scale corpus of PAN-MS image pairs (scene recognition) as well as on a benchmark hyperspectral image dataset (image classification) where we follow different experimental scenarios and find that the proposed hallucination based module indeed is capable of capturing the multi-source information, albeit the explicit absence of some of the sensor information, and aid in improved scene characterization.
LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models
Nanyang Technological University（南洋理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University of Technology（浙江工业大学）；UCAS（中国科学院大学）
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（新加坡国立大学）；National University of Defense Technology（中国人民解放军国防科技大学）；Pensees Pte Ltd（Pensees私人有限公司）；University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Recent deep learning based face recognition methods have achieved great performance, but it still remains challenging to recognize very low-resolution query face like 28x28 pixels when CCTV camera is far from the captured subject. Such face with very low-resolution is totally out of detail information of the face identity compared to normal resolution in a gallery and hard to find corresponding faces therein. To this end, we propose a Resolution Invariant Model (RIM) for addressing such cross-resolution face recognition problems, with three distinct novelties. First, RIM is a novel and unified…
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Key Laboratory of Intelligent Bionic Unmanned Systems, Ministry of Education（教育部智能仿生无人系统重点实验室）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Jilin Provincial Key Laboratory of Intelligence Science and Engineering（吉林省智能科学与工程重点实验室）；Engineering Research Center of AI and Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）
University of Southern California（南加州大学）；Facebook AI
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework…
ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Models
AI Institute, University of South Carolina（南卡罗来纳大学AI研究所）；Guru Gobind Singh Indraprastha University（古鲁·戈宾德·辛格·因德拉普拉斯塔大学）；Vellore Institute of Technology（韦洛尔理工大学）；Indian Institute of Technology (BHU)（印度理工学院（瓦拉纳西印度大学））；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）；Amazon Web Services（亚马逊网络服务）；Meta；Amazon GenAI（亚马逊生成式人工智能部门）
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficie…
Monash University（莫纳什大学）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）
Recently, a lot of automated white blood cells (WBC) or leukocyte classification techniques have been developed. However, all of these methods only utilize a single modality microscopic image i.e. either blood smear or fluorescence based, thus missing the potential of a better learning from multimodal images. In this work, we develop an efficient multimodal architecture based on a first of its kind multimodal WBC dataset for the task of WBC classification. Specifically, our proposed idea is developed in two steps - 1) First, we learn modality specific independent subnetworks inside a single ne…
Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensing
School of Computer Science, Wuhan University, China（武汉大学计算机学院）；Zhongguancun Academy, China（中关村学院）；School of Computer Science, Chongqing University, China（重庆大学计算机学院）
Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
School of Computer Science and Ningbo Institute, Northwestern Polytechnical University, China（西北工业大学计算机学院和宁波研究院）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, China（国家空天地海一体化大数据应用技术国家工程实验室）
Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation
University of Chinese Academy of Sciences（中国科学院大学）；University of Cincinnati（辛辛那提大学）；National University of Singapore（新加坡国立大学）；Beijing Jiaotong University（北京交通大学）
Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
National University of Singapore（新加坡国立大学）；University of Arizona（亚利桑那大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Skywork AI（天工智能）；Nanyang Technological University（南洋理工大学）
Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
A Survey of Multimodal Hallucination Evaluation and Detection
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Trustworthy Technology and Engineering Laboratory（可信技术与工程实验室）；Huawei（华为公司）
Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallup 学院）；Department of Electrical and Computer Engineering, McGill University（麦吉尔大学电气与计算机工程系）；School of Statistics, Renmin University of China（中国人民大学统计学院）；Tencent Jarvis Lab（腾讯 Jarvis 实验室）；Medical Artificial Intelligence Lab, Westlake University（西湖大学医学人工智能实验室）
When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
University of Trento（特伦托大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of International Relations（国际关系大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Nanjing University of Science and Technology（南京理工大学）；VCIP & TMCC & DISSec, College of Computer Science, Nankai University（南开大学计算机学院）；University of Central Florida（佛罗里达中央大学）
