用于计算的调度数据:AMD GPU上LLM训练的分层数据流水线
Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs
AI总结:
本文针对AMD GPU上LLM训练的等待时间瓶颈,利用TDM、转置加载、SDMA、RDMA等硬件特性实现分层数据流水线,提升计算利用率并获得多倍加速,相关实现已开源。
摘要:
大型语言模型(LLM)是近期机器学习突破的核心焦点。随着模型规模呈指数级增长,训练所需的计算量极为庞大。GPU可执行海量算术运算,但其计算单元就绪时所需的数据未必可用——这些数据可能仍处于慢速内存中,或存储在另一块GPU上。芯片等待期间,昂贵的计算资源处于闲置状态;大规模部署时,利用率的微小下降也会转化为巨额成本。本文将这种等待时间视为主要瓶颈,聚焦于专为显式数据移动设计的硬件特性:这些特性将数据从片外内存加载到芯片,加载过程中重排为计算所需的布局,并在GPU间传输。通过协调这些特性,我们维持流向计算单元的稳定数据流水线,确保计算单元持续处于忙碌状态。我们提出内核与框架实现,利用三条关键数据移动指令和硬件特性解决该瓶颈:张量数据移动器(TDM)将张量数据从高带宽内存(HBM)移入片上内存;转置加载指令在将张量数据加载到片上时,将其转置为LLM计算所需的布局;最后,系统DMA(SDMA)与远程DMA(RDMA)分别通过扩展网络和扩展集群网络在GPU间移动数据。结果显示,TDM和转置加载在内核层面平均分别提供1.88倍和1.19倍的加速;SDMA和RDMA在计算与通信重叠时平均分别带来1.16倍和1.33倍的加速。这是首个利用所有GPU数据流水线特性的AMD LLM训练系统,我们的实现已集成到AMD GPU的开源框架与库中。
英文摘要:
Large language models (LLM) have been the key focus of recent machine learning breakthroughs. As the model size grows exponentially, the computation needed for training is astronomical. A GPU can do a huge amount of arithmetic. However, the data its compute units need is not always available when they are ready. This data may still be in slow memory or on another GPU. While the chip waits, expensive compute sits idle, and at scale even a small drop in utilization translates into substantial cost. In this paper, we treat this waiting time as the primary bottleneck. We focus on hardware features designed for explicit data movement. These features load data from off-chip memory onto the chip, rearrange it into the layout the computation requires as it is loaded, and transfer it between GPUs. By orchestrating them, we keep a steady pipeline of data flowing to the compute units so that they remain busy. We present kernel and framework implementations that address this bottleneck using three key data-movement instructions and hardware features. The Tensor Data Mover (TDM) moves tensor data from high-bandwidth memory (HBM) into on-chip memory. Transpose-load instructions transpose tensor data into the layout that LLM computations require as it is loaded on-chip. Finally, System DMA (SDMA) and Remote DMA (RDMA) move data between GPUs over the scale-up and scale-out networks, respectively. Our results show that TDM and transpose-load provide on average 1.88x and 1.19x speedup at the kernel level, respectively. The SDMA and RDMA lead to on average 1.16x and 1.33x speedup for computation overlapped with communication. This is the first AMD LLM training system to utilize all the GPU data pipelining features. Our implementation has been integrated into open source framework and library for AMD GPUs. https://github.com/AMD-AGI/Primus-Turbo https://github.com/AMD-AGI/Primus
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。