通过自动资源感知层放置加速CNN分布式深度学习训练
Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
摘要:
卷积神经网络(CNN)模型通常用于图像分类,需要大量的训练时间来获得高精度。为此,使用多个服务器的参数服务器(PS)架构执行分布式训练。不幸的是,可扩展性已被发现是穷人在现有的架构。我们发现,PS网络是瓶颈,因为它与许多工人沟通大量的梯度和参数。这是因为在训练的每一步都必须与许多工人同步。根据模型的不同,每次同步的通信可以是几百MB。在本文中,我们提出了一个计划,以减少网络流量,通过层放置,考虑到每一层使用的资源。通过分析CNN的特点,我们发现层的放置可以以一种有效的方式进行。然后,我们将此观察结果纳入TensorFlow框架,以便可以自动放置层以进行更有效的训练。我们的评估利用这种布局方案表明,训练时间可以显着减少,而不损失许多CNN模型的准确性。
英文摘要:
The Convolutional Neural Network (CNN) model, often used for image classification, requires significant training time to obtain high accuracy. To this end, distributed training is performed with the parameter server (PS) architecture using multiple servers. Unfortunately, scalability has been found to be poor in existing architectures. We find that the PS network is the bottleneck as it communicates a large number of gradients and parameters with the many workers. This is because synchronization with the many workers has to occur at every step of training. Depending on the model, communication can be in the several hundred MBs per synchronization. In this paper, we propose a scheme to reduce network traffic through layer placement that considers the resources that each layer uses. Through analysis of the characteristics of CNN, we find that placement of layers can be done in an effective manner. We then incorporate this observation within the TensorFlow framework such that layers can be automatically placed for more efficient training. Our evaluation making use of this placement scheme show that training time can be significantly reduced without loss of accuracy for many CNN models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。