arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.05930cs.PFcs.LG
收藏

表征Alibaba-PAI上的深度学习训练工作负载

Characterizing Deep Learning Training Workloads on Alibaba-PAI

Alibaba Group(阿里巴巴集团);The University of Hong Kong(香港大学)

Mengdi Wang, Chen Meng, Guoping Long, Chuan Wu, Jun Yang, Wei Lin, Yangqing Jia

摘要:

现代深度学习模型已经在各个领域得到了利用,包括计算机视觉(CV)、自然语言处理(NLP)、搜索和推荐。在实际的人工智能集群中,训练这些模型的工作负载使用TensorFlow、Caffe、PyTorch和CNTK等软件框架运行。有效运行实际人工智能云的一个关键问题是描述这些工作负载的计算和数据传输需求,更重要的是,在给定底层软件框架和硬件配置的情况下,培训性能。在本文中,我们描述了阿里巴巴人工智能(PAI)平台的深度学习训练工作负载。我们建立了一个分析框架,调查使用不同的训练架构的各种工作负载的详细执行时间分解,以确定性能瓶颈。结果表明,训练期间的权重/梯度通信平均占用了所有工作负载总执行时间的近62%。计算部分,涉及GPU计算和内存访问,并不是基于工作负载集体行为的最大瓶颈。我们进一步评估各种潜在的软件/硬件映射的工作负载的可达到的性能,并探讨软件架构选择和硬件配置的影响。我们发现,当将60%的PS/Worker工作负载移植到AllReduce架构时,可以利用高速NVLink进行GPU互连,并且当以太网带宽从25 Gbps升级到100 Gbps时,平均可以实现1.7倍的加速。

英文摘要:

Modern deep learning models have been exploited in various domains, including computer vision (CV), natural language processing (NLP), search and recommendation. In practical AI clusters, workloads training these models are run using software frameworks such as TensorFlow, Caffe, PyTorch and CNTK. One critical issue for efficiently operating practical AI clouds, is to characterize the computing and data transfer demands of these workloads, and more importantly, the training performance given the underlying software framework and hardware configurations. In this paper, we characterize deep learning training workloads from Platform of Artificial Intelligence (PAI) in Alibaba. We establish an analytical framework to investigate detailed execution time breakdown of various workloads using different training architectures, to identify performance bottleneck. Results show that weight/gradient communication during training takes almost 62% of the total execution time among all our workloads on average. The computation part, involving both GPU computing and memory access, are not the biggest bottleneck based on collective behavior of the workloads. We further evaluate attainable performance of the workloads on various potential software/hardware mappings, and explore implications on software architecture selection and hardware configurations. We identify that 60% of PS/Worker workloads can be potentially sped up when ported to the AllReduce architecture exploiting the high-speed NVLink for GPU interconnect, and on average 1.7X speedup can be achieved when Ethernet bandwidth is upgraded from 25 Gbps to 100 Gbps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑