arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.15052cs.LGcs.AI
收藏

重新审视LLM训练中MoE与稠密模型的速度-精度对比

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training

Apple(苹果公司)

Xianzhi Du, Tom Gunter, Xiang Kong, Mark Lee, Zirui Wang, Aonan Zhang, Nan Du, Ruoming Pang

AI总结:

本文重新审视MoE与稠密LLM的对比设置,提出以步时替代FLOPs作为复杂度度量,并采用3D分片控制通信开销,实验证明MoE在速度-精度权衡上仍持续优于稠密模型。

摘要:

专家混合(MoE)通过增加模型容量同时保持计算成本不变来获得性能提升。在将MoE与稠密模型进行比较时,先前的工作通常采用以下设置:1)使用FLOPs或激活参数作为模型复杂度的度量;2)将所有模型训练到相同的token数量。我们认为这种设置有利于MoE,因为FLOPs和激活参数不能准确衡量稀疏层中的通信开销,导致MoE的实际训练预算更大。在这项工作中,我们通过采用步时(step time)作为更准确的模型复杂度度量,并在Chinchilla计算最优设置下确定总计算预算,来重新审视这些设置。为了在现代加速器上高效运行MoE,我们采用了一种3D分片方法,使稠密模型到MoE的步时增加保持在健康范围内。我们在6.4B、12.6B和29.6B三个模型规模上,评估了MoE和稠密LLM在九个0-shot和两个1-shot英语任务以及MMLU 5-shot和GSM8K 8-shot上的表现。实验结果表明,即使在这些设置下,MoE在速度-精度权衡曲线上始终以有意义的差距优于稠密LLM。我们完整的模型实现和分片策略已在~\url{https://github.com/apple/axlearn}发布。

英文摘要:

Mixture-of-Experts (MoE) enjoys performance gain by increasing model capacity while keeping computation cost constant. When comparing MoE to dense models, prior work typically adopt the following setting: 1) use FLOPs or activated parameters as a measure of model complexity; 2) train all models to the same number of tokens. We argue that this setting favors MoE as FLOPs and activated parameters do not accurately measure the communication overhead in sparse layers, leading to a larger actual training budget for MoE. In this work, we revisit the settings by adopting step time as a more accurate measure of model complexity, and by determining the total compute budget under the Chinchilla compute-optimal settings. To efficiently run MoE on modern accelerators, we adopt a 3D sharding method that keeps the dense-to-MoE step time increase within a healthy range. We evaluate MoE and dense LLMs on a set of nine 0-shot and two 1-shot English tasks, as well as MMLU 5-shot and GSM8K 8-shot across three model scales at 6.4B, 12.6B, and 29.6B. Experimental results show that even under these settings, MoE consistently outperform dense LLMs on the speed-accuracy trade-off curve with meaningful gaps. Our full model implementation and sharding strategy has been released at~\url{https://github.com/apple/axlearn}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑