Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.
Deep Multimodal Learning with Missing Modality: A Survey
The Australian National University(澳大利亚国立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Adelaide University(阿德莱德大学);The University of Surrey(萨里大学)
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)(卡内基梅隆大学信息网络研究所);School of Computer Science and Technology at Tongji University(同济大学计算机科学与技术学院)
Referring Video Object Segmentation with Cross-Modality Proxy Queries
DUT-RU International School of Information Science & Engineering, Dalian University of Technology, China(大连理工大学国际信息科学与工程学院,中国);Chinese University of Hong Kong, China(香港中文大学,中国);University of Sydney, Australia(悉尼大学,澳大利亚)
Omnimodal Dataset Distillation via High-order Proxy Alignment
University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);National University of Singapore(新加坡国立大学);The University of Sydney(悉尼大学)
Enhancing multimodal cooperation via sample-level modality valuation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Proxy-Tuning: Tailoring Multimodal Autoregressive Models for Subject-Driven Image Generation
University of Science and Technology of China(中国科学技术大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Nanjing University of Information Science and Technology(南京信息工程大学)
Buffer replay enhances the robustness of multimodal learning under missing-modality
College of Computer Science and Electronic Engineering Hunan University(湖南大学计算机科学与电子工程学院);School of Computer Science Central South University(中南大学计算机学院)
CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
Sun Yat-sen University(中山大学);The Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education(机器智能与先进计算教育部重点实验室(中山大学));JD AI Research(京东人工智能研究院)
Diagnosing and Re-learning for Balanced Multimodal Learning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Engineering Research Center of Next-Generation Search and Recommendation(下一代搜索与推荐技术工程研究中心)
LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences
Beijing University of Posts and Telecommunications(北京邮电大学);East China Normal University(华东师范大学);Shanghai University of International Business and Economics(上海对外经贸大学)
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology(华南理工大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学);Shanghai Jiao Tong University(上海交通大学)
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
Boosting Multimodal Learning via Disentangled Gradient Learning
The Laboratory of Intelligent Collaborative Computing of UESTC(UESTC智能协同计算实验室);The School of Information and Communication Engineering of UESTC(UESTC信息与通信工程学院)
Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach
Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Engineering and Technology Taxila(塔克西拉工程大学);Dalhousie University(达尔豪斯大学);Interuniversity Microelectronics Centre (IMEC)(校际微电子中心);Johannes Kepler University Linz(约翰开普勒林茨大学);Linz Institute of Technology(林茨理工学院)
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China(电子科技大学智能协同计算实验室);School of Information and Communication Engineering, University of Electronic Science and Technology of China(电子科技大学信息与通信工程学院);Peng Cheng Laboratory(鹏城实验室)
Asynchronous Multimodal Video Sequence Fusion via Learning Modality-Exclusive and -Agnostic Representations
Academy for Engineering and Technology, Fudan University(复旦大学工程技术研究院);Digital Medical Research Center, School of Basic Medical Science, Fudan University(复旦大学基础医学院数字医学研究中心);University of South Carolina(南卡罗来纳大学)
mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Tencent(腾讯);University of California, Los Angeles(加州大学洛杉矶分校);Lenovo Research(联想研究院)
Asymmetric Reinforcing against Multi-modal Representation Bias
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);The State Key Laboratory of Integrated Services Networks, Xidian University(西安电子科技大学综合业务网理论及关键技术国家重点实验室)