arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15627cs.LGcs.DC
收藏

MegaScale:将大语言模型训练扩展至超过10,000个GPU

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

ByteDance(字节跳动);Peking University(北京大学)
Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhiha… 展开作者

Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, Xin Liu

AI总结:

本文提出MegaScale生产系统,通过全栈协同设计与诊断工具,在超过一万个GPU上稳定高效地训练大语言模型,将175B模型训练的MFU提升至55.2%,比Megatron-LM提高1.34倍。

摘要:

我们介绍了构建和部署MegaScale的设计、实现与工程经验,MegaScale是一个在超过10,000个GPU规模上训练大语言模型(LLMs)的生产系统。在此规模上训练LLMs给训练效率和稳定性带来了前所未有的挑战。我们采用全栈方法,在模型块与优化器设计、计算与通信重叠、算子优化、数据流水线和网络性能调优等方面协同设计算法与系统组件。鉴于LLM训练作业持续时间很长,在整个训练过程中保持高效率(即稳定性)是生产环境中的重要考量。许多困难的稳定性问题只有在大规模下才会显现,而深入的观测能力是解决这些问题的关键。我们开发了一套诊断工具,用于监控系统栈深处的组件和事件,识别根本原因,并推导出实现容错和缓解滞后节点(stragglers)的有效技术。MegaScale在12,288个GPU上训练175B参数的LLM模型时,实现了55.2%的模型FLOPs利用率(MFU),相比Megatron-LM将MFU提升了1.34倍。我们分享了在识别和修复故障及滞后节点方面的运维经验。我们希望通过从系统视角阐述这些问题并分享经验,这项工作能够启发未来的LLM系统研究。

英文摘要:

We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑