Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert consolidation, which merges experts exhibiting similar relevance profiles across samples within each module. Extensive experiments demonstrate that CART achieves state-of-the-art performance.
Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval
NASK National Research Institute(NASK国家研究院);IDEAS Research Institute(IDEAS研究所);Warsaw University of Technology(华沙技术大学);Universitat Autonoma de Barcelona(巴塞罗那自治大学)
Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN(电子与计算机工程学院,普渡大学);Department of Computer Science, University of Exeter, UK(计算机科学系,埃克塞特大学)
Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
School of Computer Science and Statistics, Trinity College Dublin(都柏林圣三一学院计算机科学与统计学院);Department of Mathematics and Computer Science, Eindhoven University of Technology(埃因霍温理工大学数学与计算机科学系)
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Southeast University, Nanjing, China(南京东南大学);Wuhan AI Research, Wuhan, China(武汉人工智能研究院)
SwitchCIT: Switching for Continual Instruction Tuning
Coordinated Science Laboratory(协调科学实验室);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
FlyPrompt: Brain-Inspired Random-Expanded Routing with Temporal-Ensemble Experts for General Continual Learning
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University(生命科学学院,IDG/麦克格雷文脑科学研究院,清华大学);Department of Psychological and Cognitive Sciences, Tsinghua University(心理学与认知科学系,清华大学);School of Medicine, Shenzhen Campus of Sun Yat-Sen University(中山大学深圳校区医学学院)
CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs
Technology, Innovation, Entrepreneurship Department(技术、创新与创业部门);King Abdullah University of Science and Technology(卡塔尔科技大学);School of Computing and Information Systems(计算与信息系统学院);The University of Melbourne(墨尔本大学);College of Computer Science and Technology(计算机科学与技术学院);Jilin University(吉林大学);School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学);College of Computing and Data Science(计算与数据科学学院);Nanyang Technological University(南洋理工大学)
Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDP
The Hong Kong University of Science and Technology(香港科学与技术大学);Singapore Management University(新加坡管理学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Tencent AI Lab(腾讯AI实验室);University of Alberta(阿尔伯塔大学);Shanghai Jiao Tong University(上海交通大学)