arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.05962cs.DC
收藏

HiCCL:一种分层集合通信库

HiCCL: A Hierarchical Collective Communication Library

Mert Hidayetoglu, Simon Garcia de Gonzalo, Elliott Slaughter, Pinku Surana, Wen-mei Hwu, William Gropp, Alex Aiken

AI总结:

HiCCL通过组合式API将集合通信逻辑与网络优化解耦,并利用条带化和流水线优化,在Nvidia、AMD和Intel GPU平台上实现了比GPU感知MPI高17倍的平均吞吐量,同时保持跨平台可移植性。

摘要:

HiCCL(分层集合通信库)旨在应对高性能网络架构日益增长的复杂性与多样性。随着GPU系统已演变为具备不同多级通信层次的GPU网络,针对特定系统优化每一个集合函数已成为一项具有挑战性的任务。因此,许多集合通信库难以适应不同的硬件和软件,尤其是在来自不同供应商的系统之间。HiCCL的库设计通过一个组合式API将集合通信逻辑与网络特定优化解耦。通信逻辑由多播、归约和栅栏原语组合而成,然后针对指定的网络层次进行分解,在一个层次内仅使用点对点操作。最后,按照指定方式应用条带化和流水线优化以简化执行。在四台不同机器上对HiCCL进行的性能评估——两台配备Nvidia GPU,一台配备AMD GPU,一台配备Intel GPU——表明,其平均吞吐量比高度专用的GPU感知MPI实现的集合通信高出17倍,并且与供应商特定库(NCCL、RCCL和OneCCL)的吞吐量相当,同时在所有四台机器上提供了可移植性。

英文摘要:

HiCCL (Hierarchical Collective Communication Library) addresses the growing complexity and diversity in high-performance network architectures. As GPU systems have envolved into networks of GPUs with different multilevel communication hierarchies, optimizing each collective function for a specific system has become a challenging task. Consequently, many collective libraries struggle to adapt to different hardware and software, especially across systems from different vendors. HiCCL's library design decouples the collective communication logic from network-specific optimizations through a compositional API. The communication logic is composed using multicast, reduction, and fence primitives, which are then factorized for a specified network hieararchy using only point-to-point operations within a level. Finally, striping and pipelining optimizations applied as specified for streamlining the execution. Performance evaluation of HiCCL across four different machines$\unicode{x2014}$two with Nvidia GPUs, one with AMD GPUs, and one with Intel GPUs$\unicode{x2014}$demonstrates an average 17$\times$ higher throughput than the collectives of highly specialized GPU-aware MPI implementations, and competitive throughput with those of vendor-specific libraries (NCCL, RCCL, and OneCCL), while providing portability across all four machines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑