arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.04478cs.DCcs.ARcs.LGcs.NI
收藏

Themis:一种用于DL模型分布式训练的网络带宽感知集体调度策略

Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models

Georgia Institute of Technology(佐治亚理工学院);Intel(英特尔);Meta

Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Sridharan, Tushar Krishna

摘要:

分布式训练是通过将任务拆分到多个NPU(例如,GPU/TPU)。然而,分布式训练增加了NPU之间的通信开销,以便根据并行化策略来同步梯度和/或激活。在下一代大规模培训平台中,NPU将通过多维网络连接,具有不同的异构带宽。这项工作确定了一个迫在眉睫的挑战,保持所有的网络维度繁忙,最大限度地提高网络带宽的混合环境中,如果我们利用调度技术的集体通信系统今天。我们提出了Themis,一种新的集体调度方案,动态调度集体(分为块),以平衡所有维度的通信负载,进一步提高网络带宽利用率。我们的研究结果表明,平均而言,Themis可以将单个All-Reduce的网络带宽利用率提高1.72倍(最大2.70倍),并将ResNet-152、GNMT、DLRM和Transformer-1 T等真实工作负载的端到端训练迭代性能提升1.49倍(2.25 X最大值)、1.30 X(1.78 X最大值)、1.30 X(1.77 X最大值)和1.25 X(1.53 X最大值)。

英文摘要:

Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (e.g., GPU/TPU). However, distributed training adds communication overhead between the NPUs in order to synchronize the gradients and/or activation, depending on the parallelization strategy. In next-generation platforms for training at scale, NPUs will be connected through multi-dimensional networks with diverse, heterogeneous bandwidths. This work identifies a looming challenge of keeping all network dimensions busy and maximizing the network BW within the hybrid environment if we leverage scheduling techniques for collective communication on systems today. We propose Themis, a novel collective scheduling scheme that dynamically schedules collectives (divided into chunks) to balance the communication loads across all dimensions, further improving the network BW utilization. Our results show that on average, Themis can improve the network BW utilization of the single All-Reduce by 1.72X (2.70X max), and improve the end-to-end training iteration performance of real workloads such as ResNet-152, GNMT, DLRM, and Transformer-1T by 1.49X (2.25X max), 1.30X (1.78X max), 1.30X (1.77X max), and 1.25X (1.53X max), respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑