GRIN: GRadient-INformed MoE（梯度感知混合专家模型）
GRIN: GRadient-INformed MoE
AI总结：
针对MoE模型离散专家路由阻碍梯度优化的问题，提出GRIN梯度感知训练方法，结合稀疏梯度估计与防令牌丢弃的并行策略，构建的小激活参数MoE模型性能匹配更大规模稠密模型，多基准测试表现优异。
摘要：
混合专家（Mixture-of-Experts, MoE）模型通过专家路由实现稀疏计算，仅选择性激活少量专家模块，因此比稠密模型具备更高效的扩展能力。然而，稀疏计算对传统训练范式构成挑战：离散的专家路由阻碍了标准反向传播，进而影响了作为深度学习基石的基于梯度的优化。为充分挖掘MoE的扩展潜力，我们提出GRIN（GRadient-INformed MoE training，梯度感知MoE训练方法），该方法引入稀疏梯度估计用于专家路由，并配置模型并行机制以避免令牌丢弃。将GRIN应用于自回归语言建模，我们构建了一个top-2的16×3.8B MoE模型。该模型仅激活66亿参数，性能优于70亿参数的稠密模型，且与在相同数据上训练的140亿参数稠密模型表现相当。在多种任务上的广泛评估表明，GRIN可显著提升MoE的效能，在MMLU上达到79.4分，HellaSwag上83.7分，HumanEval上74.4分，MATH上58.9分。
英文摘要：
Mixture-of-Experts (MoE) models scale more effectively than dense models due to sparse computation through expert routing, selectively activating only a small subset of expert modules. However, sparse computation challenges traditional training practices, as discrete expert routing hinders standard backpropagation and thus gradient-based optimization, which are the cornerstone of deep learning. To better pursue the scaling power of MoE, we introduce GRIN (GRadient-INformed MoE training), which incorporates sparse gradient estimation for expert routing and configures model parallelism to avoid token dropping. Applying GRIN to autoregressive language modeling, we develop a top-2 16$\times$3.8B MoE model. Our model, with only 6.6B activated parameters, outperforms a 7B dense model and matches the performance of a 14B dense model trained on the same data. Extensive evaluations across diverse tasks demonstrate the potential of GRIN to significantly enhance MoE efficacy, achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
