综合最优集体算法
Synthesizing Optimal Collective Algorithms
摘要:
集体通信算法是分布式计算的重要组成部分。事实上,在深度学习的情况下,集体通信是数据并行训练的Amdahl瓶颈。本文介绍了SCCL(合成集体通信库),一个系统的方法来合成集体通信算法,显式定制到一个特定的硬件拓扑结构。SCCL沿着帕累托边界合成算法,从延迟最优到带宽最优的集体实现。本文演示了如何将SCCL的综合编码为一个无量词的SMT公式,该公式可以输出到定理证明器。我们进一步展示了如何利用拓扑和集体的对称性来扩展我们的合成。我们合成和介绍新的延迟和带宽优化算法在两个流行的硬件拓扑结构的文献中没有看到。我们还展示了SCCL如何有效地降低算法实现两个硬件架构(NVIDIA和AMD),并展示了竞争力的性能与手工优化的集体通信库。
英文摘要:
Collective communication algorithms are an important component of distributed computation. Indeed, in the case of deep-learning, collective communication is the Amdahl's bottleneck of data-parallel training. This paper introduces SCCL (for Synthesized Collective Communication Library), a systematic approach to synthesize collective communication algorithms that are explicitly tailored to a particular hardware topology. SCCL synthesizes algorithms along the Pareto-frontier spanning from latency-optimal to bandwidth-optimal implementations of a collective. The paper demonstrates how to encode SCCL's synthesis as a quantifier-free SMT formula which can be discharged to a theorem prover. We further demonstrate how to scale our synthesis by exploiting symmetries in topologies and collectives. We synthesize and introduce novel latency and bandwidth optimal algorithms not seen in the literature on two popular hardware topologies. We also show how SCCL efficiently lowers algorithms to implementations on two hardware architectures (NVIDIA and AMD) and demonstrate competitive performance with hand optimized collective communication libraries.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。