动态专家特化：迈向无灾难性遗忘的多领域MoE适配
Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
AI总结：
本文提出DES-MoE框架，通过自适应路由器蒸馏、实时专家-领域关联映射和三阶段自适应微调，实现多领域MoE适配中的动态专家特化，在六个领域上显著减少灾难性遗忘并加快收敛。
摘要：
Mixture-of-Experts (MoE)模型通过稀疏门控的专家子网络提供了巨大的容量，然而在不发生灾难性遗忘的情况下将其适配到多个领域仍然是一个未解决的挑战。现有方法要么计算开销过高，要么遭受跨领域干扰，要么需要为每个领域单独运行。我们提出DES-MoE，一个用于Mixture-of-Experts模型多领域适配的动态专家特化框架。DES-MoE通过三项创新来解决灾难性遗忘问题：（1）一个自适应路由器，通过蒸馏平衡预训练知识保留与任务特定更新；（2）实时专家-领域关联映射，以隔离领域特定梯度；（3）一个三阶段自适应微调调度，逐步冻结非特化参数。在六个领域（数学、代码、法律等）上的评估表明，DES-MoE在训练一个统一模型的同时达到了与单领域ESFT相当的性能，在领域数量从2扩展到6的过程中，与全量微调相比将遗忘减少了89%，并且比传统方法收敛速度提高了68%。我们的工作确立了动态专家隔离作为多任务MoE适配的可扩展范式。
英文摘要：
Mixture-of-Experts (MoE) models offer immense capacity via sparsely gated expert subnetworks, yet adapting them to multiple domains without catastrophic forgetting remains an open challenge. Existing approaches either incur prohibitive computation, suffer cross-domain interference, or require separate runs per domain. We propose DES-MoE, a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models. DES-MoE addresses catastrophic forgetting through three innovations: (1) an adaptive router balancing pre-trained knowledge retention and task-specific updates via distillation, (2) real-time expert-domain correlation mapping to isolate domain-specific gradients, and (3) a three-phase adaptive fine-tuning schedule that progressively freezes non-specialized parameters. Evaluated on six domains (math, code, law, etc.), DES-MoE matches single-domain ESFT performance while training one unified model, reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6, and achieves 68% faster convergence than conventional methods. Our work establishes dynamic expert isolation as a scalable paradigm for multi-task MoE adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
