Fusing data from multiple modalities provides more information to train machine learning systems. However, it is prohibitively expensive and time-consuming to label each modality with a large amount of data, which leads to a crucial problem of semi-supervised multi-modal learning. Existing methods suffer from either ineffective fusion across modalities or lack of theoretical guarantees under proper assumptions. In this paper, we propose a novel information-theoretic approach, namely \textbf{T}otal \textbf{C}orrelation \textbf{G}ain \textbf{M}aximization (TCGM), for semi-supervised multi-modal learning, which is endowed with promising properties: (i) it can utilize effectively the information across different modalities of unlabeled data points to facilitate training classifiers of each modality (ii) it has theoretical guarantee to identify Bayesian classifiers, i.e., the ground truth posteriors of all modalities. Specifically, by maximizing TC-induced loss (namely TC gain) over classifiers of all modalities, these classifiers can cooperatively discover the equivalent class of ground-truth classifiers; and identify the unique ones by leveraging limited percentage of labeled data. We apply our method to various tasks and achieve state-of-the-art results, including news classification, emotion recognition and disease prediction.
Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning
Siebel School of Computing and Data Science, UIUC, US（UIUC计算机与数据科学学院）；Department of Electrical and Computer Engineering, UIUC, US（UIUC电气与计算机工程学院）；VinUni-Illinois Smart Health Center, VinUniversity, Vietnam（Vin大学-伊利诺伊智能健康中心）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；National Institute on Aging, National Institute of Health（美国国立卫生研究院国家老龄化研究所）
Enhancing multimodal cooperation via sample-level modality valuation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）
Mind the Gap: Promoting Missing Modality Brain Tumor Segmentation with Alignment
School of Robotics Xi’an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi'an Jiaotong-Liverpool University Suzhou, China；School of Advanced Technology Xi’an Jiaotong-Liverpool University Suzhou, China；Data Science Research Center Duke Kunshan University Suzhou, China
Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation
Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Liverpool（利物浦大学）；AI 3 Fudan University（复旦大学）；Duke Kunshan University（杜克-昆山大学）；Zhongshan Hospital（中山医院）
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Department of Computer Engineering, Ankara University（安卡拉大学计算机工程系）；Department of Modeling and Simulation, Graduate School of Informatics, METU（信息学院建模与模拟系，METU）
Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Control Science and Engineering, Dalian University of Technology（大连理工大学控制科学与工程学院）
Balancing Multimodal Training Through Game-Theoretic Regularization
Department of Electrical Engineering, KU Leuven（KU Leuven 电子工程系）；Department of Development and Regeneration, KU Leuven（KU Leuven 发展与再生系）；Media Lab and EECS, MIT（MIT 媒体实验室和电子工程与计算机科学系）
Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Shenzhen Audencia Financial Technology Institute, Shenzhen University（深圳大学深圳Audencia金融科技研究院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；CATL（宁德时代）；School of Computing and Artificial Intelligence, Fuyao University of Science and Technology（福耀科技大学计算与人工智能学院）；School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）
RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of Target Cognition and Application Technology(TCAT), Aerospace Information Research Institute, Chinese Academy of Sciences（目标认知与应用技术重点实验室）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；Peng Cheng Laboratory, Shenzhen（鹏城实验室）
