arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.06533cs.DCcs.AI
收藏

利用计算和通信任务的智能并行性加速分布式K-FAC

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

The Hong Kong University of Science and Technology(香港科技大学)

Shaohuai Shi, Lin Zhang, Bo Li

摘要:

在GPU集群上使用同步随机梯度下降(SGD)的分布式训练已被广泛用于加速深度模型的训练过程。然而,SGD仅在模型参数更新中利用一阶梯度,这可能需要数天或数周。最近的研究已经成功地利用近似二阶信息来加速训练过程,其中Kronecker因子近似曲率(KFAC)成为训练深度模型的最有效的近似算法之一。然而,当利用GPU集群来训练具有分布式KFAC(D-KFAC)的模型时,其在每次迭代期间引起大量计算以及引入额外通信。在这项工作中,我们提出了D-KFAC(SPD-KFAC)与智能并行的计算和通信任务,以减少迭代时间。具体而言,1)我们首先描述了D-KFAC的性能瓶颈,2)我们设计并实现了一个流水线机制的克罗内克因子计算和通信与动态张量融合,3)我们开发了一个负载均衡布局,在GPU集群上的多个矩阵求逆。我们在一个64 GPU的集群上进行了实际的实验,该集群具有100 Gb/s的InfiniBand互连。实验结果表明,我们提出的SPD-KFAC训练方案可以实现10%-35%的国家的最先进的算法的改进。

英文摘要:

Distributed training with synchronous stochastic gradient descent (SGD) on GPU clusters has been widely used to accelerate the training process of deep models. However, SGD only utilizes the first-order gradient in model parameter updates, which may take days or weeks. Recent studies have successfully exploited approximate second-order information to speed up the training process, in which the Kronecker-Factored Approximate Curvature (KFAC) emerges as one of the most efficient approximation algorithms for training deep models. Yet, when leveraging GPU clusters to train models with distributed KFAC (D-KFAC), it incurs extensive computation as well as introduces extra communications during each iteration. In this work, we propose D-KFAC (SPD-KFAC) with smart parallelism of computing and communication tasks to reduce the iteration time. Specifically, 1) we first characterize the performance bottlenecks of D-KFAC, 2) we design and implement a pipelining mechanism for Kronecker factors computation and communication with dynamic tensor fusion, and 3) we develop a load balancing placement for inverting multiple matrices on GPU clusters. We conduct real-world experiments on a 64-GPU cluster with 100Gb/s InfiniBand interconnect. Experimental results show that our proposed SPD-KFAC training scheme can achieve 10%-35% improvement over state-of-the-art algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑