SAME: 用于多模态持续指令调优的稳定混合专家模型
SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
AI总结：
针对多模态持续指令调优中专家路由漂移和专家漂移问题，提出稳定混合专家模型（SAME），通过正交子空间分解路由动态和曲率感知缩放更新专家，实现无重放的状态最优性能。
摘要：
多模态大语言模型（MLLMs）通过指令调优实现了强大的性能，但实际部署需要它们持续扩展能力，这使得多模态持续指令调优（MCIT）变得至关重要。最近的方法利用稀疏专家路由来促进任务专业化，但我们发现专家路由过程会随着数据分布的演变而发生漂移。例如，之前激活定位专家的接地查询在学习OCR任务后可能被路由到不相关的专家。同时，与接地相关的专家可能被新任务覆盖而失去原有功能。这种失败反映了两个问题：路由器漂移（专家选择随时间变得不一致）和专家漂移（共享专家跨任务被覆盖）。因此，我们提出了用于MCIT的稳定混合专家模型（SAME）。为了解决路由器漂移，SAME通过将路由动态分解为正交子空间并仅更新任务相关方向来稳定专家选择。为了缓解专家漂移，我们通过使用历史输入协方差进行曲率感知缩放来调节专家更新，无需重放。SAME还引入了自适应专家激活，在训练期间冻结选中的专家，减少冗余计算和跨任务干扰。我们还引入了一个新的基准来评估长任务序列的MCIT，大量实验证明了SAME的最优性能。代码可在 https://github.com/LAMDA-CL/Prism 获取。
英文摘要：
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. We also introduce a new benchmark to evaluate MCIT with long task sequence, and extensive experiments demonstrate SAME's SOTA performance. Code is available at https://github.com/LAMDA-CL/Prism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
