Drop-Upcycling:通过部分重新初始化训练稀疏混合专家模型
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
AI总结:
针对 Upcycling 初始化 MoE 后长期训练缓慢的问题,提出部分权重重初始化的 Drop-Upcycling,以促进专家专门化,并以约四分之一训练 FLOPs 使 5.9B 活跃参数 MoE 媲美 13B 稠密模型。
摘要:
与具有等效容量的稠密模型相比,混合专家(MoE)架构显著降低了训练和推理成本。Upcycling 是一种利用预训练稠密模型来初始化并训练 MoE 模型的方法。尽管 upcycling 能带来初始性能提升,但其训练进展比从头训练更慢,长期来看会导致次优性能。我们提出 Drop-Upcycling,一种有效解决该问题的方法。Drop-Upcycling 结合了两种看似矛盾的途径:利用预训练稠密模型的知识,同时对部分权重进行统计意义上的重新初始化。该方法有策略地促进专家专门化,显著提升 MoE 模型获取知识的效率。大规模实验表明,Drop-Upcycling 在长期训练中显著优于以往的 MoE 构建方法,尤其是在数千亿或更多 token 上训练时。因此,我们拥有 5.9B 活跃参数的 MoE 模型达到了与同一模型家族中 13B 稠密模型相当的性能,而所需训练 FLOPs 约为其 1/4。包括源代码、训练数据、模型检查点和日志在内的所有实验资源均公开提供,以促进可复现性和未来关于 MoE 的研究。
英文摘要:
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that initializes and trains an MoE model using a pre-trained dense model. While upcycling leads to initial performance gains, the training progresses slower than when trained from scratch, leading to suboptimal performance in the long term. We propose Drop-Upcycling - a method that effectively addresses this problem. Drop-Upcycling combines two seemingly contradictory approaches: utilizing the knowledge of pre-trained dense models while statistically re-initializing some parts of the weights. This approach strategically promotes expert specialization, significantly enhancing the MoE model's efficiency in knowledge acquisition. Extensive large-scale experiments demonstrate that Drop-Upcycling significantly outperforms previous MoE construction methods in the long term, specifically when training on hundreds of billions of tokens or more. As a result, our MoE model with 5.9B active parameters achieves comparable performance to a 13B dense model in the same model family, while requiring approximately 1/4 of the training FLOPs. All experimental resources, including source code, training data, model checkpoints and logs, are publicly available to promote reproducibility and future research on MoE.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。