HiCCL (Hierarchical Collective Communication Library) addresses the growing complexity and diversity in high-performance network architectures. As GPU systems have envolved into networks of GPUs with different multilevel communication hierarchies, optimizing each collective function for a specific system has become a challenging task. Consequently, many collective libraries struggle to adapt to different hardware and software, especially across systems from different vendors. HiCCL's library design decouples the collective communication logic from network-specific optimizations through a compositional API. The communication logic is composed using multicast, reduction, and fence primitives, which are then factorized for a specified network hieararchy using only point-to-point operations within a level. Finally, striping and pipelining optimizations applied as specified for streamlining the execution.
Performance evaluation of HiCCL across four different machines$\unicode{x2014}$two with Nvidia GPUs, one with AMD GPUs, and one with Intel GPUs$\unicode{x2014}$demonstrates an average 17$\times$ higher throughput than the collectives of highly specialized GPU-aware MPI implementations, and competitive throughput with those of vendor-specific libraries (NCCL, RCCL, and OneCCL), while providing portability across all four machines.
An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
Infrawaves;Shanghai Innovation Institute(上海创新研究院);Beihang University(北京航空航天大学);Tsinghua University(清华大学);Zhipu AI(智谱AI);China Unicom Research Institute(中国联通研究院);Shanghai AI Power Technology Co., Ltd(上海人工智能功率科技有限公司)
Comprehensive Deadlock Prevention for GPU Collective Communication
School of Computer Science, Peking University(北京大学计算机学院);OneFlow Research(OneFlow研究院);National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院并行与分布式计算国家重点实验室)
StrataCL: Fabric-Native Communication Library for Production Supernodes
Peking University(北京大学);SKLP, Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Technologies Company Ltd.(华为技术有限公司);Shanghai Jiao Tong University(上海交通大学);Beijing Tongming Lake Information Technology Application Innovation Center, China(北京通明湖信息技术应用创新中心)
针对分布式 AI 工作负载的通信瓶颈,提出零冗余的 fabric 原生通信库 StrataCL,通过分配时注册等技术优化,在华为 CloudMatrix384 上显著提升了带宽与多类工作负载的性能。
Tools-CC-Bench: a Benchmark Suite for Collective Communication with Compression in HPC and AI Workloads
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机学院);School of Computer Science, Nanjing University(南京大学计算机系)
Exploiting network topology in brain-scale simulations of spiking neural networks
Institute for Advanced Simulation (IAS-6), Jülich Research Centre(先进模拟研究所(IAS-6),于利希研究中心);RWTH Aachen University(亚琛工业大学);Department of Physics, Faculty 1, RWTH Aachen University(亚琛工业大学第一学院物理系);Department of Psychiatry, Psychotherapy and Psychosomatics, School of Medicine, RWTH Aachen University(亚琛大学医学院精神病学、心理治疗与身心医学科);Neuromorphic Software Ecosystems (PGI-15), Jülich Research Centre(神经形态软件生态系统(PGI-15),于利希研究中心);JARA BRAIN Institute Structure-Function Relationships (INM-10), Jülich Research Centre(JARA大脑结构-功能关系研究所(INM-10),于利希研究中心)