arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.02752cs.DCcs.AIcs.LG
收藏

基于PaddlePaddle的端到端自适应分布式训练

End-to-end Adaptive Distributed Training on PaddlePaddle

Baidu Inc.(百度公司);Peking University(北京大学)

Yulong Ao, Zhihua Wu, Dianhai Yu, Weibao Gong, Zhiqing Kui, Minxu Zhang, Zilingfeng Ye, Liang Shen, Yanjun Ma, Tian Wu, Haifeng Wang, Wei Zeng, Chao Yang

摘要:

分布式训练已成为一种普遍而有效的训练大型神经网络(NN)模型处理大量数据的方法。然而,它是非常具有挑战性的,以满足各种神经网络模型的要求,不同的计算资源,以及它们在训练工作中的动态变化。在这项研究中,我们设计了我们的分布式培训框架在一个系统的端到端的观点,提供内置的自适应能力,为不同的场景,特别是工业应用和生产环境,充分考虑资源分配,模型划分,任务放置,和分布式执行。基于统一的分布式图和统一的集群对象,我们的自适应框架配备了一个全局成本模型和一个全局规划器,它可以实现任意并行,资源感知的布局,多模式执行,容错和弹性分布式训练。实验表明,该框架能够满足应用程序的多样性和资源的异构性的各种需求,具有很强的竞争力的性能。拥有2600亿个参数的ERNIE语言模型在数千个AI处理器上高效训练,具有91.7%的弱可扩展性。通过采用异构流水线异步执行,推荐系统的模型的吞吐量可以分别增加到仅GPU和仅CPU训练的2.1倍和3.3倍。此外,容错和弹性的分布式训练已成功地应用于在线工业应用程序,这给长期训练失败的工作数量减少了34.49%,并提高了33.91%的生产环境中的全局调度效率。

英文摘要:

Distributed training has become a pervasive and effective approach for training a large neural network (NN) model with processing massive data. However, it is very challenging to satisfy requirements from various NN models, diverse computing resources, and their dynamic changes during a training job. In this study, we design our distributed training framework in a systematic end-to-end view to provide the built-in adaptive ability for different scenarios, especially for industrial applications and production environments, by fully considering resource allocation, model partition, task placement, and distributed execution. Based on the unified distributed graph and the unified cluster object, our adaptive framework is equipped with a global cost model and a global planner, which can enable arbitrary parallelism, resource-aware placement, multi-mode execution, fault-tolerant, and elastic distributed training. The experiments demonstrate that our framework can satisfy various requirements from the diversity of applications and the heterogeneity of resources with highly competitive performance. The ERNIE language model with 260 billion parameters is efficiently trained on thousands of AI processors with 91.7% weak scalability. The throughput of the model from the recommender system by employing the heterogeneous pipeline asynchronous execution can be increased up to 2.1 times and 3.3 times that of the GPU-only and CPU-only training respectively. Moreover, the fault-tolerant and elastic distributed training have been successfully applied to the online industrial applications, which give a reduction of 34.49% in the number of failed long-term training jobs and an increase of 33.91% for the global scheduling efficiency in the production environment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑