TACCL:使用通信草图指导集体算法合成
TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
摘要:
机器学习模型越来越多地在多个GPU和服务器上进行训练。在这种情况下,数据使用AlltoAll和AllReduce等通信集体在GPU之间传输,这可能成为训练大型模型的重要瓶颈。因此,重要的是使用有效的算法进行集体通信。我们开发TACCL,一个工具,使算法设计人员能够引导合成器自动生成算法,为给定的硬件配置和通信集体。TACCL使用一种新颖的通信草图抽象来从设计者那里获得关键信息,以显著减少搜索空间并引导合成器朝向更好的算法。TACCL还使用了一种新的问题编码,使其能够扩展到单节点拓扑之外。我们使用TACCL合成算法的三个集体和两个硬件拓扑结构:DGX-2和NDv 2。我们证明了TACCL合成的算法比Nvidia集体通信库(NCCL)的性能高出6.7倍。我们还表明,TACCL可以将Transformer-XL和BERT模型的端到端训练速度提高11%--对于不同的批量大小,提高2.3倍。
英文摘要:
Machine learning models are increasingly being trained across multiple GPUs and servers. In this setting, data is transferred between GPUs using communication collectives such as AlltoAll and AllReduce, which can become a significant bottleneck in training large models. Thus, it is important to use efficient algorithms for collective communication. We develop TACCL, a tool that enables algorithm designers to guide a synthesizer into automatically generating algorithms for a given hardware configuration and communication collective. TACCL uses a novel communication sketch abstraction to get crucial information from the designer to significantly reduce the search space and guide the synthesizer towards better algorithms. TACCL also uses a novel encoding of the problem that allows it to scale beyond single-node topologies. We use TACCL to synthesize algorithms for three collectives and two hardware topologies: DGX-2 and NDv2. We demonstrate that the algorithms synthesized by TACCL outperform the Nvidia Collective Communication Library (NCCL) by up to 6.7x. We also show that TACCL can speed up end-to-end training of Transformer-XL and BERT models by 11%--2.3x for different batch sizes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。