arXiv 2601.130202026-07-20cs.LGcs.AI

PASs-MoE：通过路径激活子空间减轻路由器与专家之间的错位协同漂移以进行持续学习

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Southeast University, Nanjing, China（南京东南大学）；Wuhan AI Research, Wuhan, China（武汉人工智能研究院）

Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang

备注：: Published in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), Volume 1: Long Papers. 14 pages. Code is available at https://github.com/yueluoshuangtian/PASs-MoE
发表信息：: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31959--31972, San Diego, California, United States, July 2026. Association for Computational Linguistics
DOI：: 10.18653/v1/2026.acl-long.1474

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究持续指令调整中多模态大语言模型的问题，提出基于路径激活子空间的固定容量PASs - MoE - LoRA方法，含PAS引导的重新加权和PAS感知的秩稳定，实验表明该方法在准确性和抗遗忘性上优于基线和变体且不增参数。

摘要：

持续指令调整（CIT）要求多模态大语言模型（MLLMs）适应一系列任务且不遗忘先前能力。常见策略是通过将输入路由到不同的LoRA专家来隔离更新。然而，现有的基于LoRA的专家混合（MoE）方法通常以不加区分的方式联合更新路由器和专家，导致路由器偏好与专家适应路径协同漂移，偏离早期输入 - 专家专业化，即错位协同漂移，这模糊了专家职责并加剧遗忘。为解决此问题，我们引入路径激活子空间（PASs），它反映输入在每个专家中激活的低秩路径方向，为路由和保存提供能力对齐的坐标系。基于PASs，我们提出基于固定容量PASs的MoE - LoRA方法，包括PAS引导的重新加权和PAS感知的秩稳定。在CIT基准测试中的实验表明，我们的方法在准确性和抗遗忘性方面均优于一系列传统持续学习基线和MoE - LoRA变体，且不增加模型参数。代码可公开获取。

英文摘要：

Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input--expert specialization. We term this as Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting. To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE--LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE--LoRA variants in both accuracy and resistance to forgetting, without increasing model parameters. Our code is publicly available at https://github.com/yueluoshuangtian/PASs-MoE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。