Lazarus：混合专家模型的弹性与韧性训练
Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models
AI总结：
针对MoE模型训练中故障频繁且现有方案缺乏弹性的问题，提出Lazarus系统，通过自适应专家副本分配和最优放置算法，实现故障后充分利用节点，性能提升最高达5.7倍。
摘要：
稀疏激活的混合专家（MoE）架构已被越来越多地用于进一步扩展大型语言模型（LLM）。然而，随着训练规模的扩大，频繁的故障仍然构成重大挑战。即使一次故障的代价也是巨大的，因为所有GPU都需要空闲等待直到故障解决，并且由于训练必须从检查点重新开始，可能会损失大量的训练进度。这个问题因在公共云上越来越多地使用竞价实例进行模型训练而加剧，尽管竞价实例提供了可观的成本节省，但引入了频繁的抢占——本质上是在整个训练过程中定期发生的故障。现有的高效容错训练解决方案要么缺乏弹性，要么依赖于在流水线并行中构建韧性，由于MoE架构采用的专家并行策略，这些方法无法应用于MoE模型。我们提出了Lazarus，一个用于MoE模型弹性与韧性训练的系统。Lazarus自适应地分配专家副本以解决专家工作负载中固有的不平衡并加速训练，同时开发了一种可证明最优的专家放置算法，以最大化故障后恢复的概率。通过自适应的专家放置和灵活的分发器，Lazarus还可以在故障后充分利用所有可用节点，不留下任何GPU空闲。我们的评估表明，在频繁节点故障下，Lazarus的性能比现有MoE训练系统高出5.7倍，在真实的竞价实例轨迹上高出3.4倍。
英文摘要：
Sparsely-activated Mixture-of-Experts (MoE) architecture has increasingly been adopted to further scale large language models (LLMs). However, frequent failures still pose significant challenges as training scales. The cost of even a single failure is significant, as all GPUs need to idle wait until the failure is resolved, potentially losing considerable training progress as training has to restart from checkpoints. This problem is exacerbated by the growing use of spot instances on public clouds for model training, which despite offering substantial cost savings, introduce frequent preemptions-essentially failures that regularly occur throughout the training process. Existing solutions for efficient fault-tolerant training either lack elasticity or rely on building resiliency into pipeline parallelism, which cannot be applied to MoE models due to the expert parallelism strategy adopted by the MoE architecture. We present Lazarus, a system for resilient and elastic training of MoE models. Lazarus adaptively allocates expert replicas to address the inherent imbalance in expert workload and speeds up training, while a provably optimal expert placement algorithm is developed to maximize the probability of recovery upon failures. Through adaptive expert placement and a flexible token dispatcher, Lazarus can also fully utilize all available nodes after failures, leaving no GPU idle. Our evaluation shows that Lazarus outperforms existing MoE training systems by up to 5.7x under frequent node failures and 3.4x on a real spot instance trace.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
