The human language can be expressed through multiple sources of information known as modalities, including tones of voice, facial gestures, and spoken language. Recent multimodal learning with strong performances on human-centric tasks such as sentiment analysis and emotion recognition are often black-box, with very limited interpretability. In this paper we propose Multimodal Routing, which dynamically adjusts weights between input modalities and output representations differently for each input sample. Multimodal routing can identify relative importance of both individual modalities and cross-modality features. Moreover, the weight assignment by routing allows us to interpret modality-prediction relationships not only globally (i.e. general trends over the whole dataset), but also locally for each single input sample, meanwhile keeping competitive performance compared to state-of-the-art methods.
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
Xi’an Jiaotong University（西安交通大学）；ARC Lab, Tencent IEG（腾讯IEG ARC实验室）；City University of Hong Kong（香港城市大学）；Institute of Automation, CAS（中国科学院自动化研究所）；Harvard University（哈佛大学）；Nanjing University（南京大学）
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
School of Computer Science, Peking University（北京大学计算机科学学院）；MOE Key Laboratory of Computational Linguistics, Peking University（北京大学教育部计算语言学重点实验室）；National Key Laboratory for Multimedia Information Processing, Peking University（北京大学国家多媒体信息处理重点实验室）
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）；Renmin University of China（中国人民大学）
Training-Free Multimodal Large Language Model Orchestration
Media Analytics and Computing Lab, Xiamen University, Xiamen, China（厦门大学媒体分析与计算实验室）；Institute of Artificial Intelligence, Xiamen University, Xiamen, China（厦门大学人工智能研究院）；School of Informatics, Xiamen University, Xiamen, China（厦门大学信息学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Department of Computer Science, National University of Singapore, Singapore（新加坡国立大学计算机科学系）
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
School of Mechanical, Aerospace, and Manufacturing Engineering, University of Connecticut, Storrs, CT 06269（机械、航空航天与制造工程学院，康涅狄格大学，斯托尔斯，CT 06269）；Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA（机械工程系，麻省理工学院，剑桥，MA 02139，美国）
SAE-V: Interpreting Multimodal Models for Enhanced Alignment
Institute for AI, Peking University, Beijing, China（人工智能研究院，北京大学，北京，中国）；State Key Laboratory of General Artificial Intelligence, Institute for AI, Peking University, Beijing, China（通用人工智能国家重点实验室，人工智能研究院，北京大学，北京，中国）
