Nexus：专业化与适应性相结合以高效训练混合专家模型
Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
AI总结：
提出增强型MoE架构Nexus，通过自适应路由将稠密专家模型升级回收，实现高效专业化并支持灵活添加新专家，在初始升级和扩展新专家上分别获得2.1%和18.8%的相对增益。
摘要：
效率、专业化以及对新数据分布的适应性，是当前大型语言模型中难以兼得的特性。混合专家架构因其固有的条件计算能力能够实现这些理想特性，而成为重要研究焦点。在本研究中，我们专注于将稠密专家模型“升级回收”为 MoE，旨在提升专业化的同时，也增加轻松适应新任务的能力。我们提出了 Nexus，一种带有自适应路由的增强型 MoE 架构，模型通过学习从领域表示中投影专家嵌入。该方法使 Nexus 能够在初始升级回收后，通过单独训练的稠密模型灵活添加新专家，无需为未见数据域进行大规模 MoE 训练。实验表明，Nexus 在初始升级回收上较基线实现高达 2.1% 的相对增益，并通过使用有限微调数据扩展 MoE 新专家时获得 18.8% 的相对增益。Nexus 的这种灵活性对于构建开源生态系统至关重要，使每个用户都能根据自身需求持续组装专属的 MoE 混合模型。
英文摘要：
Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models. The Mixture of Experts (MoE) architecture has been the focus of significant research because its inherent conditional computation enables such desirable properties. In this work, we focus on "upcycling" dense expert models into an MoE, aiming to improve specialization while also adding the ability to adapt to new tasks easily. We introduce Nexus, an enhanced MoE architecture with adaptive routing where the model learns to project expert embeddings from domain representations. This approach allows Nexus to flexibly add new experts after the initial upcycling through separately trained dense models, without requiring large-scale MoE training for unseen data domains. Our experiments show that Nexus achieves a relative gain of up to 2.1% over the baseline for initial upcycling, and a 18.8% relative gain for extending the MoE with a new expert by using limited finetuning data. This flexibility of Nexus is crucial to enable an open-source ecosystem where every user continuously assembles their own MoE-mix according to their needs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
