DisDP:用于模型分片数据并行训练的计算、网络和存储解耦
DisDP: Disaggregating Compute, Network, and Storage for Model-Sharded Data-Parallel Training
AI总结:
研究针对模型分片数据并行训练中网络通信开销大问题,提出DisDP架构,通过将计算、网络和存储解耦,把集合操作卸载到智能网卡和交换机,存储卸载到增强型参数服务器,提升了GPU利用率,实验验证了该方法的高效性。
摘要:
模型分片数据并行(MSDP),如ZeRO,将模型状态均匀分片到所有GPU上,因其低GPU内存容量需求而被广泛应用于大语言模型预训练。然而,MSDP会因额外的网络通信集合操作带来严重开销。本文提出DisDP,一种完全解耦的分布式数据并行架构,它将计算、网络和存储完全解耦,使GPU仅专注计算部分以最大化利用率。关键在于将集合操作完全卸载到智能网卡和智能交换机,避免矩阵乘法内核与集合内核干扰,将存储完全卸载到智能交换机增强的参数服务器,实现单个参数服务器为大量工作节点提供线性可扩展服务。在8个分布式GPU上训练175B模型时,DisDP比现有最先进训练系统性能提升3.98倍,验证了解耦的效率。
英文摘要:
Model-sharded data parallelism (MSDP), e.g., ZeRO, evenly shards the model states across all GPUs, and thus has been widely adopted by LLM pre-training, such as Llama and DeepSeek, due to its low GPU memory capacity requirement. However, MSDP introduces severe overhead from additional network communication collectives (i.e., AllGather and ReduceScatter). Although the collectives themselves only occupy fewer than 10% of GPU SMs, their execution time increases by 41% due to the serial execution of aggregated CPU/GPU-managed compute (i.e., GEMM), network (i.e., NCCL), and storage (i.e., optimizer states). To this end, we present DisDP, a fully disaggregated distributed data-parallel architecture that first fully disaggregates compute, network, and storage for MSDP, such that GPUs only focus on the computing part, and thus the GPU utilization is maximized. The key idea is 1) fully offloading collectives to SmartNICs and SmartSwitch to avoid interference between GEMM kernels and collective kernels, and 2) fully offloading storage to a SmartSwitch-enhanced parameter server that allows a single PS to serve massive workers with linear scalability. DisDP on 8 distributed GPUs outperforms the state-of-the-art training systems by 3.98x when training on a 175B model, validating the efficiency of disaggregation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。