无需训练的动态专家语言模型再利用
Training-Free Dynamic Upcycling of Expert Language Models
AI总结：
本文提出DUME方法，通过动态添加不同领域训练的密集专家，构建统一的MoE模型，无需额外训练即可保持原有能力，在语言建模和推理任务中表现优异。
摘要：
大型语言模型（LLMs）在多种专业任务上表现出色，但训练成本高且缺乏领域专业知识。专家微调可解决此问题，但易导致过度专业化。本文提出动态再利用MoE（DUME），通过重用不同领域训练的密集专家构建统一模型，无需额外训练。DUME利用岭回归的闭式解，实现动态添加专家并保持模型性能。在因果语言建模和推理任务中，DUME均优于基线方法，且可进一步微调提升性能。
英文摘要：
Large Language Models (LLMs) have achieved remarkable performance on a wide range of specialized tasks, exhibiting strong problem-solving capabilities. However, training these models is prohibitively expensive, and they often lack domain-specific expertise because they rely on general knowledge datasets. Expertise finetuning can address this issue; however, it often leads to overspecialization, and developing a single multi-domain expert remains difficult due to diverging objectives. Furthermore, multitask training is challenging due to interference and catastrophic forgetting. Existing work proposes combining the expertise of dense models within a Mixture of Experts (MoE) architecture, although this approach still requires multitask finetuning. To address these issues, we introduce Dynamic Upcycling MoE (DUME), a novel approach that reuses dense experts trained on different domains to construct a unified MoE model. Our method builds a single multitask model that preserves the capabilities of the original dense experts without requiring additional training. DUME is both cost-efficient and scalable: by leveraging the closed-form solution of ridge regression, it eliminates the need for further optimization and enables experts to be added dynamically while maintaining the model's original performance. We demonstrate that DUME consistently outperforms baseline approaches in both causal language modeling and reasoning settings. Finally, we also show that the DUME model can be fine-tuned to further improve performance. We show that, in the causal language modeling setting, DUME can retain up to 97.6% of a dense expert model specialized in one particular domain, and that it can also surpass it in the reasoning setting, where it can achieve 102.1% of the dense expert performance. Our code is available at: github.com/gensyn-ai/dume.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
