Swing：用于更高带宽 Allreduce 的环形捷径方法
Swing: Short-cutting Rings for Higher Bandwidth Allreduce
AI总结：
针对torus网络中通信距离大导致allreduce带宽低的问题，提出通过方向切换保持低通信距离的Swing算法，在多种拓扑下对32B到128MiB向量的性能最高超现有算法3倍。
摘要：
Allreduce（全归约）集合操作在分布式系统运行的工作负载总运行时间中占比很高。决定其性能的一个因素是通信节点间的距离，尤其在torus（环面）这类网络中，距离越大意味着同一链路上需要转发多条消息，进而降低allreduce带宽。Torus网络被广泛用于面向机器学习工作负载优化的系统（例如Google TPU和Amazon Trainium设备），以及部分Top500超级计算机。为提升torus网络上的allreduce性能，我们提出了Swing算法，该算法通过在torus的不同方向间切换，保持通信节点间的低距离。我们的分析与实验评估表明，在不同类型的torus及类torus拓扑上，无论拓扑的形状和规模如何，对于32B到128MiB大小的向量，Swing的性能均优于现有allreduce算法，最高可达3倍。
英文摘要：
The allreduce collective operation accounts for a significant fraction of the runtime of workloads running on distributed systems. One factor determining its performance is the distance between communicating nodes, especially on networks like torus, where a higher distance implies multiple messages being forwarded on the same link, thus reducing the allreduce bandwidth. Torus networks are widely used on systems optimized for machine learning workloads (e.g., Google TPUs and Amazon Trainium devices), as well as on some of the Top500 supercomputers. To improve allreduce performance on torus networks we introduce Swing, a new algorithm that keeps a low distance between communicating nodes by swinging between torus directions. Our analysis and experimental evaluation show that Swing outperforms by up to 3x existing allreduce algorithms for vectors ranging from 32B to 128MiB, on different types of torus and torus-like topologies, regardless of their shape and size.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
