分布式大语言模型训练与推理的性能建模和工作负载分析
Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference
AI总结:
本文提出一个分析框架,用于建模分布式LLM训练与推理的性能,并分析技术扩展下瓶颈演变,为未来系统设计提供指导。
摘要:
将未来系统设计与大语言模型(LLM)日益增长的计算需求对齐,无疑是当今世界的一个重要问题。在此,我们通过一个分析框架,提出了一种通用的性能建模方法和分布式LLM训练与推理的工作负载分析,该框架准确考虑了计算、内存子系统、网络以及各种并行化策略(模型并行、数据并行、流水线并行和序列并行)。我们使用文献和相关行业供应商(如NVIDIA)的已发表数据验证了我们的性能预测。对于分布式训练,我们研究了不同激活重计算方法下LLM的内存占用,剖析了从A100到B200(约35倍加速,紧密跟随NVIDIA的扩展趋势)巨大性能提升背后的关键因素,并进一步在不同技术节点(12纳米至1纳米)进行了设计空间探索,以研究逻辑、内存和网络扩展对性能的影响。对于推理,我们在矩阵乘法级别分析了不同GPU系统上不同操作的计算与内存受限性,并进一步探讨了DRAM内存技术扩展对推理延迟的影响。利用我们的建模框架,我们揭示了随着技术扩展,LLM训练和推理性能瓶颈的演变,从而为设计未来用于LLM训练和推理的系统提供了见解。
英文摘要:
Aligning future system design with the ever-increasing compute needs of large language models (LLMs) is undoubtedly an important problem in today's world. Here, we propose a general performance modeling methodology and workload analysis of distributed LLM training and inference through an analytical framework that accurately considers compute, memory sub-system, network, and various parallelization strategies (model parallel, data parallel, pipeline parallel, and sequence parallel). We validate our performance predictions with published data from literature and relevant industry vendors (e.g., NVIDIA). For distributed training, we investigate the memory footprint of LLMs for different activation re-computation methods, dissect the key factors behind the massive performance gain from A100 to B200 ($\sim$ 35x speed-up closely following NVIDIA's scaling trend), and further run a design space exploration at different technology nodes (12 nm to 1 nm) to study the impact of logic, memory, and network scaling on the performance. For inference, we analyze the compute versus memory boundedness of different operations at a matrix-multiply level for different GPU systems and further explore the impact of DRAM memory technology scaling on inference latency. Utilizing our modeling framework, we reveal the evolution of performance bottlenecks for both LLM training and inference with technology scaling, thus, providing insights to design future systems for LLM training and inference.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。