Missing modalities present a fundamental challenge in multimodal models, often causing catastrophic performance degradation. Our observations suggest that this fragility stems from an imbalanced learning process, where the model develops an implicit preference for certain modalities, leading to the under-optimization of others. We propose a simple yet efficient method to address this challenge. The central insight of our work is that the dominance relationship between modalities can be effectively discerned and quantified in the frequency domain. To leverage this principle, we first introduce a Frequency Ratio Metric (FRM) to quantify modality preference by analyzing features in the frequency domain. Guided by FRM, we then propose a Multimodal Weight Allocation Module, a plug-and-play component that dynamically re-balances the contribution of each branch during training, promoting a more holistic learning paradigm. Extensive experiments demonstrate that MWAM can be seamlessly integrated into diverse architectural backbones, such as those based on CNNs and ViTs. Furthermore, MWAM delivers consistent performance gains across a wide range of tasks and modality combinations. This advancement extends beyond merely optimizing the performance of the base model; it also manifests as further performance improvements to state-of-the-art methods addressing the missing modality problem.
MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering
School of Computer and Information Science, Qinghai Institute of Technology（青海理工学院计算机与信息科学学院）；School of Information Science and Technology, Harbin Institute of Technology (Shen Zhen)（哈尔滨工业大学（深圳）信息科学与技术学院）；School of Computer Science and Technology, Qinghai University（青海大学计算机科学与技术学院）；Eryuan Digital Technology Co., Ltd.（洱源数字科技有限公司）
MGML: A Plug-and-Play Meta-Guided Multi-Modal Learning Framework for Incomplete Multimodal Brain Tumor Segmentation
Nanchang University（南昌大学）；Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University（中山大学孙逸仙纪念医院）；Zhujiang Hospital, Southern Medical University（南方医科大学珠江医院）；University of California Los Angeles（加州大学洛杉矶分校）；Hubei University of Economics（湖北经济学院）；Xi'an Jiaotong Liverpool University（西交利物浦大学）；Southeast University（东南大学）
Plug-and-Adapt: Multimodal Coreference Resolution at First Sight with a Pretrained Alignment Model
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能全国重点实验室）；Centre for Frontier AI Research and Institute of High-Performance Computing, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心与高性能计算研究所）
National Yang Ming Chiao Tung University（国立阳明交通大学）；Google（谷歌）
In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire mod…
MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中国教育部，厦门大学）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）
From Image to Video, what do we need in multimodal LLMs?
Intelligent Computing and Machine Learning Lab, School of ASEE, Beihang University（北京航空航天大学自动化学院智能计算与机器学习实验室）；Xiaohongshu（小红书）；School of Sino-French Engineer, Beihang University（北京航空航天大学中法工程师学院）；College of Engineering and Computer Science, VinUniversity（Vin大学工程与计算机科学学院）
Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Baidu Inc（百度公司）
Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Great Bay University（大湾区大学）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University（深圳大学）
Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
Department of Informatics University of Bern（伯尔尼大学信息学院）；College of Computing and Data Science Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Software Engineering Xi’an Jiaotong University（西安交通大学软件工程学院）
School of Computer Science, South China Normal University（南方科技大学计算机科学学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Computer Science, City University of Hong Kong（香港城市大学计算机科学学院）
