稠密反向传播改进稀疏混合专家模型的训练
Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
AI总结：
针对MoE稀疏反向更新导致训练不稳定的问题，提出Default MoE方法，用专家输出的指数移动平均替代缺失激活，实现稠密梯度更新，显著提升训练性能且无显著计算开销。
摘要：
混合专家（MoE）预训练比稠密 Transformer 预训练更具可扩展性，因为 MoE 学会将输入路由到其稀疏的前馈参数中。然而，这意味着 MoE 仅接收稀疏的反向更新，导致训练不稳定和性能次优。我们提出了一种轻量级近似方法，在继续稀疏激活其参数的同时，为 MoE 路由器提供稠密的梯度更新。我们的方法被称为 Default MoE，它用默认输出替代缺失的专家激活，该默认输出由训练过程中先前观察到的专家输出的指数移动平均组成。这使得路由器能够为每个 token 接收来自每个专家的信号，从而显著提升训练性能。我们的 Default MoE 在多种设置下均优于标准 TopK 路由，且无需显著的计算开销。代码：https://github.com/vatsal0/default-moe。
英文摘要：
Mixture of Experts (MoE) pretraining is more scalable than dense Transformer pretraining, because MoEs learn to route inputs to a sparse set of their feedforward parameters. However, this means that MoEs only receive a sparse backward update, leading to training instability and suboptimal performance. We present a lightweight approximation method that gives the MoE router a dense gradient update while continuing to sparsely activate its parameters. Our method, which we refer to as Default MoE, substitutes missing expert activations with default outputs consisting of an exponential moving average of expert outputs previously seen over the course of training. This allows the router to receive signals from every expert for each token, leading to significant improvements in training performance. Our Default MoE outperforms standard TopK routing in a variety of settings without requiring significant computational overhead. Code: https://github.com/vatsal0/default-moe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
