A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., 90% training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal learning with missing modality in terms of flexibility (missing modalities in training, testing, or both) and efficiency (most training data have incomplete modality). Technically, we propose a new method named SMIL that leverages Bayesian meta-learning in uniformly achieving both objectives. To validate our idea, we conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI, and avMNIST. The results prove the state-of-the-art performance of SMIL over existing methods and generative baselines including autoencoders and generative adversarial networks. Our code is available at https://github.com/mengmenm/SMIL.
Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China.（智能与计算学院，天津大学，天津300350，中国）；Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong.（统计与精算系，计算与数据科学学院，香港大学，香港）；Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China.（放射科，天津华和医院，天津300350，中国）；Tianjin Key Laboratory of Cerebral Vascular and Neurodegenerative Diseases, Tianjin 300350, China.（天津脑血管与神经退行性疾病重点实验室，天津300350，中国）；Medical School of Tianjin University, Tianjin 300072, China.（天津大学医学院，天津300072，中国）
CCSD: Cross-Modal Compositional Self-Distillation for Robust Brain Tumor Segmentation with Missing Modalities
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（上海智能自主系统研究院，同济大学）；College of Biomedical Engineering, Fudan University（生物医学工程学院，复旦大学）；School of Medicine, Tongji University（医学院，同济大学）；College of Information Engineering, China Jiliang University（信息工程学院，中国嘉兴大学）
Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement
State Key Laboratory of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；National Computer Network Emergency Response Technical Team, Coordination Center of China（中国国家计算机网络应急技术处理协调中心）
CausalDisenSeg: A Causality-Guided Disentanglement Framework with Counterfactual Reasoning for Robust Brain Tumor Segmentation Under Missing Modalities
School of Information Engineering, Nanchang University（南昌大学信息工程学院）；School of Mathematics and Computer Sciences, Nanchang University（南昌大学数学与计算机科学学院）
Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data
Queensland Brain Institute, The University of Queensland（昆士兰脑研究所，昆士兰大学）；School of Information Technology and Electrical Engineering, The University of Queensland（信息技术与电气工程学院，昆士兰大学）
SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models
State Key Laboratory of Robotics and Intelligent Systems（机器人学国家重点实验室）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Shanghai Jiao Tong University（上海交通大学）
CMRxRecon2024: A Multi-Modality, Multi-View K-Space Dataset Boosting Universal Machine Learning for Accelerated Cardiac MRI
Xiamen University（厦门大学）；Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Fudan University（复旦大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；University of Shanghai for Science and Technology（上海理工大学）；GE Healthcare（通用电气医疗集团）；Siemens Healthineers Ltd.（西门子医疗有限公司）；The Hong Kong Polytechnic University（香港理工大学）；Pontificia Universidad Católica de Chile（智利天主教大学）
ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations
Sichuan University（四川大学）；Zhejiang University（浙江大学）；Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）；Lenovo Group Limited（联想集团有限公司）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education（教育部机器学习与工业智能工程研究中心）
MS-Mix: Sentiment-Guided Adaptive Augmentation for Multimodal Sentiment Analysis
Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；School of Engineering, Westlake University（西湖大学工学院）
