We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research.
Kernel machines that adapt to GPUs for effective large batch training
Ohio State University(俄亥俄州立大学)
现代机器学习模型通常使用随机梯度下降(SGD)在诸如GPU的大规模并行计算资源上进行训练。增加小批量是利用并行计算能力的一种简单而直接的方法。对于小批量,批量大小的增加会导致训练时间的成比例减少,这种现象称为线性缩放。但是,将批大小增加到超过某个值不会进一步改善训练时间。在本文中,我们开发的第一个分析框架,扩展线性缩放,以匹配并行计算能力的资源。该框架是为一类经典的内核机器设计的。它自动修改标准内核机器以输出数学上等效的预测函数,但允许扩展的线性缩放,即,在给定硬件上实现更高效的并行化和更快的训练时间。由此产生的算法是准确的,原则性和非常快。例如,使用单个Titan Xp GPU,在ImageNet上使用$1.3\times 10^6$数据点和$1000$标签进行训练需要不到一个小时,而较小的数据集(如MNIST)需要几秒钟。由于参数是基于理论界解析地选择的,因此除了选择核和核参数之外,几乎不需要调整,进一步促进了这些方法的实际使用。
PGT-I: Scaling Spatiotemporal GNNs with Memory-Efficient Distributed Training
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Argonne National Laboratory(阿贡国家实验室);Arizona State University(亚利桑那州立大学);Sandia National Laboratories(桑塔纳国家实验室)
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Université Lumière Lyon 2(里昂第二大学);CNRS(法国国家科学研究中心);Ecole Centrale de Lyon(里昂中央理工学院);INSA Lyon(里昂国立应用科学学院);Universite Claude Bernard Lyon 1(克洛德·贝尔纳里昂第一大学);LIRIS, UMR5205(里昂信息图像与科学计算实验室,UMR5205)