HetPipe：通过集成流水线模型并行和数据并行，在（不稳定的）异构GPU集群上实现大型DNN训练
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
摘要：
深度神经网络（DNN）模型的规模不断增长，以提高模型的准确性和质量。此外，对于大型DNN模型的训练，由于新GPU架构的发布周期较短，使用异构GPU是不可避免的。在本文中，我们研究了如何在异构GPU集群上实现大型DNN模型的训练，该集群可能包括独立的无法用于训练的古怪GPU。我们提出了一个DNN训练系统HetPipe（Heterogeneous Pipeline），它集成了流水线模型并行（PMP）和数据并行（DP）。在HetPipe中，一组多个GPU（称为虚拟工作器）以流水线方式处理小批，多个这样的虚拟工作器采用数据并行以获得更高的性能。我们还提出了一种新的参数同步模型，我们称之为波同步并行（WSP），以适应PMP和DP的虚拟工作者，并提供收敛证明WSP。我们在给定的异构设置上的实验结果表明，与最先进的DP技术相比，使用HetPipe，DNN模型的收敛速度快了49%。
英文摘要：
Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
