Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning
Siebel School of Computing and Data Science, UIUC, US（UIUC计算机与数据科学学院）；Department of Electrical and Computer Engineering, UIUC, US（UIUC电气与计算机工程学院）；VinUni-Illinois Smart Health Center, VinUniversity, Vietnam（Vin大学-伊利诺伊智能健康中心）
Duy A. Nguyen, Abhi Kamboj, Minh N. Do
备注：
Accepted and presented at IJCAI 2025 in Montreal, Canada
Addressing missing modalities and limited labeled data is crucial for advancing robust multimodal learning. We propose Robult, a scalable framework designed to mitigate these challenges by preserving modality-specific information and leveraging redundancy through a novel information-theoretic approach. Robult optimizes two core objectives: (1) a soft Positive-Unlabeled (PU) contrastive loss that maximizes task-relevant feature alignment while effectively utilizing limited labeled data in semi-supervised settings, and (2) a latent reconstruction loss that ensures unique modality-specific information is retained. These strategies, embedded within a modular design, enhance performance across various downstream tasks and ensure resilience to incomplete modalities during inference. Experimental results across diverse datasets validate that Robult achieves superior performance over existing approaches in both semi-supervised learning and missing modality contexts. Furthermore, its lightweight design promotes scalability and seamless integration with existing architectures, making it suitable for real-world multimodal applications.
Adaptive Redundancy Regulation for Balanced Multimodal Information Refinement
Department of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术系，哈尔滨工业大学）；Harbin Institute of Technology Zhengzhou Research Institute（哈尔滨工业大学郑州研究所）；Harbin Institute of Technology Suzhou Research Institute（哈尔滨工业大学苏州研究所）；School of Mathematical Sciences, University of Electronic Science and Technology of China（数学学院，电子科学与技术大学）；School of Electronic Engineering and Computer Science, Institute of Digital Media, Peking University（电子工程与计算机科学系，数字媒体研究所，北京大学）
Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）
Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned…
For better user experience and business effectiveness, Click-Through Rate (CTR) prediction has been one of the most important tasks in E-commerce. Although extensive CTR prediction models have been proposed, learning good representation of items from multimodal features is still less investigated, considering an item in E-commerce usually contains multiple heterogeneous modalities. Previous works either concatenate the multiple modality features, that is equivalent to giving a fixed importance weight to each modality; or learn dynamic weights of different modalities for different items through…
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Shaping Initial State Prevents Modality Competition in Multi-modal Fusion: A Two-stage Scheduling Framework via Fast Partial Information Decomposition
Institute of Medical Technology, Peking University Health Science Center（北京大学人民医院医学技术研究所）；National Institute of Health Data Science（国家健康数据科学研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）
Peking University（北京大学）；Nanyang Technological University（南洋理工大学）；University of Technology, Sydney（悉尼科技大学）；Hefei Institute of Intelligent Machines, Chinese Academy of Sciences（中国科学院合肥智能机械研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
The features used in many image analysis-based applications are frequently of very high dimension. Feature extraction offers several advantages in high-dimensional cases, and many recent studies have used multi-task feature extraction approaches, which often outperform single-task feature extraction approaches. However, most of these methods are limited in that they only consider data represented by a single type of feature, even though features usually represent images from multiple modalities. We therefore propose a novel large margin multi-modal multi-task feature extraction (LM3FE) framewo…
EchoBridge: Long-Tail-Aware ECG-Echocardiography Text Alignment for Echocardiography-Derived Cardiac Findings
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；University of the Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能重点实验室）；National Institute of Health Data Science, Peking University（北京大学健康数据科学研究所）；Department of Cardiology, Peking University People’s Hospital（北京大学人民医院心内科）
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition
University of Electronic Science and Technology of China（电子科技大学）；Defense Innovation Institute, Academy of Military Sciences（国防科技创新研究院）；Peking University（北京大学）；High-tech Institute（高新技术研究所）
Indian Institute of Technology Indore（印度理工学院印多尔分校）；University of Manitoba（曼尼托巴大学）
Machine learning models have been successfully employed in the diagnosis of Schizophrenia disease. The impact of classification models and the feature selection techniques on the diagnosis of Schizophrenia have not been evaluated. Here, we sought to access the performance of classification models along with different feature selection approaches on the structural magnetic resonance imaging data. The data consist of 72 subjects with Schizophrenia and 74 healthy control subjects. We evaluated different classification algorithms based on support vector machine (SVM), random forest, kernel ridge r…
