arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.06324cs.DCcs.SYeess.SY
收藏

面向GPU集体通信的全面死锁预防

Comprehensive Deadlock Prevention for GPU Collective Communication

School of Computer Science, Peking University(北京大学计算机学院);OneFlow Research(OneFlow研究院);National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院并行与分布式计算国家重点实验室)

Lichen Pan, Juncheng Liu, Yongquan Fu, Jinhui Yuan, Rongkai Zhang, Pengze Li, Zhen Xiao

AI总结:

本文提出DFCCL库,通过底层预emption机制有效防止GPU集体通信死锁,同时保持高性能,实验显示其在多种场景下有效,性能不逊于NCCL。

摘要:

分布式深度神经网络训练需要高效的GPU集体通信,而此类通信本身易产生死锁。在分布式深度学习应用中,多个集体通信操作相互等待导致死锁的问题尤为突出。GPU集体死锁对分布式深度学习的正确性和效率构成重大挑战,目前尚无通用有效的解决方案。只有在特定场景下,通过让应用程序在不同GPU间保持集体操作顺序一致性,才能避免循环依赖和死锁。本文提出DFCCL,一种新型GPU集体通信库,提供全面的GPU集体死锁预防方法,同时保持高性能。DFCCL在底层库级别实现了GPU集体的抢占机制,即使应用程序导致循环依赖,也能有效防止死锁。DFCCL通过集体操作的执行和调度方法保证高性能。实验表明,DFCCL在多种情况下有效防止GPU集体死锁。此外,广泛评估显示,DFCCL的性能与或优于NCCL,即高度优化于NVIDIA GPU的当前最先进的集体通信库。

英文摘要:

Distributed deep neural network training necessitates efficient GPU collective communications, which are inherently susceptible to deadlocks. GPU collective deadlocks arise easily in distributed deep learning applications when multiple collectives circularly wait for each other. GPU collective deadlocks pose a significant challenge to the correct functioning and efficiency of distributed deep learning, and no general effective solutions are currently available. Only in specific scenarios, ad-hoc methods, making an application invoke collectives in a consistent order across GPUs, can be used to prevent circular collective dependency and deadlocks. This paper presents DFCCL, a novel GPU collective communication library that provides a comprehensive approach for GPU collective deadlock prevention while maintaining high performance. DFCCL achieves preemption for GPU collectives at the bottom library level, effectively preventing deadlocks even if applications cause circular collective dependency. DFCCL ensures high performance with its execution and scheduling methods for collectives. Experiments show that DFCCL effectively prevents GPU collective deadlocks in various situations. Moreover, extensive evaluations demonstrate that DFCCL delivers performance comparable to or superior to NCCL, the state-of-the-art collective communication library highly optimized for NVIDIA GPUs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑