面向分布式AI的网络卸载带宽最优广播与全收集
Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI
AI总结:
针对FSDP训练中集合操作竞争注入带宽的问题,提出利用硬件多播的带宽最优全收集算法,实现2倍流量减少,并采用智能网卡卸载以扩展到1.6 Tbit/s链路。
摘要:
在完全分片数据并行(FSDP)训练流程中,集合操作可以交错执行,以最大化通信与计算的重叠。在此场景下,诸如全收集(Allgather)和归约散播(Reduce-Scatter)等未完成的操作可能会竞争注入带宽,并产生流水线气泡。为解决此问题,我们提出了一种新颖的带宽最优的全收集集合算法,该算法利用硬件多播。我们使用多播构建了一个恒定时间的可靠广播协议,作为构建最优全收集调度方案的基石。我们的全收集算法在188节点测试平台上实现了2倍的流量减少。为使主机端免于运行该协议,我们采用了智能网卡(SmartNIC)卸载。我们提取了全收集算法中的并行性,并将其映射到专门用于隐藏数据移动成本的智能网卡上。我们展示了我们的智能网卡卸载集合进度引擎能够扩展到下一代1.6 Tbit/s链路。
英文摘要:
In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstanding operations such as Allgather and Reduce-Scatter can compete for the injection bandwidth and create pipeline bubbles. To address this problem, we propose a novel bandwidth-optimal Allgather collective algorithm that leverages hardware multicast. We use multicast to build a constant-time reliable Broadcast protocol, a building block for constructing an optimal Allgather schedule. Our Allgather algorithm achieves 2x traffic reduction on a 188-node testbed. To free the host side from running the protocol, we employ SmartNIC offloading. We extract the parallelism in our Allgather algorithm and map it to a SmartNIC specialized for hiding the cost of data movement. We show that our SmartNIC-offloaded collective progress engine can scale to the next generation of 1.6 Tbit/s links.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。