In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire model. We further explore the effect of different prompt configurations and analyze the robustness to missing modality. Extensive experiments are conducted to show the effectiveness of our prompt learning framework that improves the performance under various missing-modality cases, while alleviating the requirement of heavy model re-training. Code is available.
Synergistic Prompting for Robust Visual Recognition with Missing Modalities
Beijing Institute of Technology（北京理工大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Nanjing University of Science and Technology（南京理工大学）；National University of Singapore（新加坡国立大学）；Systems Laboratory of Anhui Province, Hefei University of Technology（安徽省系统实验室，合肥工业大学）；National Innovative Institute of Defense Technology（国家创新防御技术研究院）；Intelligent Science & Technology Academy of CASIC（中国航天科技集团智能科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology Taxila（塔克西拉工程大学）；Dalhousie University（达尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心）；Johannes Kepler University Linz（约翰开普勒林茨大学）；Linz Institute of Technology（林茨理工学院）
Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
IMT School for Advanced Studies of Lucca（卢卡高等研究院）；Institute of Computational Perception, Johannes Kepler University（约翰开普勒大学计算感知研究所）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Engineering and Technology, Taxila（塔克西拉工程技术大学）；Dalhousie University（戴尔豪斯大学）；Interuniversity Microelectronics Centre (IMEC)（微电子学研究中心）；Linz Institute of Technology（林茨理工学院）
Robust Multi-Modal Face Anti-Spoofing with Domain Adaptation: Tackling Missing Modalities, Noisy Pseudo-Labels, and Model Degradation
College of Computer and Cyber Security, Fujian Normal University（计算机与网络安全部，福建师范大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China（东部理工学院计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University, China（上海大学机电工程与自动化学院）
CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Aberdeen（阿伯丁大学）
Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
School of Mathematics, Shandong University（山东大学数学学院）；GSFS, The University of Tokyo（东京大学GSFS）；MedVisAI Lab（医学视觉人工智能实验室）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光前医学院）；Centre of AI in Medicine, Singapore（新加坡医学人工智能中心）
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China)（青岛软件研究所，计算机科学与技术学院，中国石油大学（华东））；Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software（山东省智能油气工业软件重点实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
University of Electronic Science and Technology of China（电子科技大学）；Intelligent Digital Media Technology Key Laboratory of Sichuan Province（四川省智能数字媒体技术重点实验室）
PRA-PoE: Robust Multimodal Alzheimer's Diagnosis with Arbitrary Missing Modalities
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China（生物医学工程系，香港理工大学，香港特别行政区，中国）；Department of Technology Management for Innovation, The University of Tokyo, Japan（创新技术管理系，东京大学，日本）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China（数据科学与人工智能系，香港理工大学，香港特别行政区，中国）
Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning
Johannes Kepler University（约翰内斯·开普勒大学）；Aalto University（阿alto大学）；University of Michigan-Flint（密歇根大学弗林特分校）；Macquarie University（麦考瑞大学）；Institute of Computational Perception, Johannes Kepler University Linz（计算感知研究所，约翰内斯·开普勒大学林茨）；Linz Institute of Technology（林茨技术研究所）
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore（新加坡国立大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；National Neuroscience Institute（国家神经科学研究所）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）
Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
University of Aberdeen（阿伯丁大学）；NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息技术研究所）；West Virginia University（西弗吉尼亚大学）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））
ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Lenovo Ltd.（联想有限公司）；Duke University（杜克大学）；University of North Charlotte（北卡罗来纳大学夏洛特分校）
