Skywork-MoE：深入探究混合专家语言模型的训练技术
Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
AI总结：
本文提出 Skywork-MoE，一个基于 Skywork-13B 稠密检查点向上循环训练的 1460 亿参数混合专家语言模型，并通过门控 logit 归一化和自适应辅助损失系数提升专家多样化与训练效果，在多个基准上取得优异表现。
摘要：
在本技术报告中，我们介绍了在开发 Skywork-MoE 过程中实施的训练方法。Skywork-MoE 是一个高性能的混合专家（MoE）大语言模型（LLM），拥有 1460 亿参数和 16 个专家。它从我们已有的 Skywork-13B 模型的稠密检查点初始化。我们探索了向上循环（upcycling）初始化与从头训练初始化的相对效果。我们的研究结果表明，在这两种方法之间进行选择时，应同时考虑现有稠密检查点的性能和 MoE 训练预算。我们重点介绍了两项创新技术：门控 logit 归一化，可改善专家多样化；以及自适应辅助损失系数，允许对辅助损失系数进行逐层调整。我们的实验结果验证了这些方法的有效性。利用这些技术和见解，我们在 SkyPile 语料库的一个精简子集上训练了向上循环得到的 Skywork-MoE。评估结果表明，我们的模型在广泛的基准测试中表现出强劲的性能。
英文摘要：
In this technical report, we introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is initialized from the pre-existing dense checkpoints of our Skywork-13B model. We explore the comparative effectiveness of upcycling versus training from scratch initializations. Our findings suggest that the choice between these two approaches should consider both the performance of the existing dense checkpoints and the MoE training budget. We highlight two innovative techniques: gating logit normalization, which improves expert diversification, and adaptive auxiliary loss coefficients, allowing for layer-specific adjustment of auxiliary loss coefficients. Our experimental results validate the effectiveness of these methods. Leveraging these techniques and insights, we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile corpus. The evaluation results demonstrate that our model delivers strong performance across a wide range of benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
