面向直连拓扑的高效全对全集合通信调度
Efficient All-to-All Collective Communication Schedules for Direct-Connect Topologies
AI总结：
本文针对超级计算机规模直连互连，采用整体方法优化全对全集合通信性能，解决了为任意拓扑开发高效带宽最优调度并适配多种运行时的挑战，并提出一种接近最优性能的新型拓扑。
摘要：
全对全集合通信原语广泛应用于机器学习（ML）和高性能计算（HPC）负载，优化其性能是ML和HPC社区共同关注的焦点。全对全是一项极具挑战性的负载，在规模化时会严重消耗底层互连带宽。本文采用整体性方法，优化超级计算机规模直连互连上的全对全集合通信性能。我们解决了若干算法与实际挑战，包括为任意拓扑开发高效且带宽最优的全对全调度，以及将调度适配至各种运行时和互连技术。我们还提出了一种新型拓扑，能提供接近最优的全对全性能。
英文摘要：
The all-to-all collective communications primitive is widely used in machine learning (ML) and high performance computing (HPC) workloads, and optimizing its performance is of interest to both ML and HPC communities. All-to-all is a particularly challenging workload that can severely strain the underlying interconnect bandwidth at scale. This paper takes a holistic approach to optimize the performance of all-to-all collective communications on supercomputer-scale direct-connect interconnects. We address several algorithmic and practical challenges in developing efficient and bandwidth-optimal all-to-all schedules for any topology and lowering the schedules to various runtimes and interconnect technologies. We also propose a novel topology that delivers near-optimal all-to-all performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
