arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.07974cs.DCcs.AI
收藏

CrossoverScheduler:以交叉方式重叠多个分布式训练应用

CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner

University of Torontoland(多伦多大学(Torontoland));Googol ShallowMind(谷歌浅思(Googol ShallowMind))

Cheng Luo, Lei Qu, Youshan Miao, Peng Cheng, Yongqiang Xiong

摘要:

分布式深度学习工作负载包括GPU集群上的吞吐量密集型训练任务,其中分布式随机梯度下降(SGD)在反向传播后会导致显著的通信延迟,迫使工作人员通过集中式参数服务器或直接在分散式工作人员中等待梯度同步。我们提出了Crossoverlapping,一种算法,使通信周期的分布式训练应用程序通过流水线通信和计算被其他应用程序填充。通过交叉验证,可以在不牺牲收敛速度和网络精度的情况下显著提高分布式训练的运行性能。我们通过引入交叉同步来实现这一点,交叉同步允许多个分布式深度学习应用程序交替地共享同一GPU。Crossovercraft的原型被构建并与Horovod集成。在多种分布式任务上的实验表明,Crossoverride在ImageNet数据集上对图像分类任务实现了20%的\times加速比。

英文摘要:

Distributed deep learning workloads include throughput-intensive training tasks on the GPU clusters, where the Distributed Stochastic Gradient Descent (SGD) incurs significant communication delays after backward propagation, forces workers to wait for the gradient synchronization via a centralized parameter server or directly in decentralized workers. We present CrossoverScheduler, an algorithm that enables communication cycles of a distributed training application to be filled by other applications through pipelining communication and computation. With CrossoverScheduler, the running performance of distributed training can be significantly improved without sacrificing convergence rate and network accuracy. We achieve so by introducing Crossover Synchronization which allows multiple distributed deep learning applications to time-share the same GPU alternately. The prototype of CrossoverScheduler is built and integrated with Horovod. Experiments on a variety of distributed tasks show that CrossoverScheduler achieves 20% \times speedup for image classification tasks on ImageNet dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑