通过稀疏插值混合专家实现大语言模型升级改造中的自动专家发现
Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
AI总结：
本研究提出SIMoE指令微调算法，可将稠密预训练LLM自动转化为多专业领域的MoE模型，在指定稀疏约束下识别结构稀疏的领域专家并学习输入依赖的专家合并策略，在常用基准上实现SOTA性能与最优性能-计算权衡。
摘要：
我们提出稀疏插值混合专家（Sparse Interpolated Mixture-of-Experts，SIMoE）指令微调方法，这是一种端到端算法，旨在将稠密预训练大语言模型（LLM）微调为具备多个专业领域能力的MoE风格模型。在指令微调过程中，SIMoE在指定的稀疏性约束下自动识别多个专业专家，每个专家代表种子LLM参数的结构稀疏子集，对应数据中的领域特定知识。SIMoE还通过路由网络同时学习依赖输入的专家合并策略，利用丰富的跨专家知识实现更优的下游泛化能力，效果超越现有基线。实验表明，SIMoE在常用指令微调基准上持续取得最优性能，同时与所有基线相比保持了最优的性能-计算权衡。
英文摘要：
We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
