深度神经网络的分布式训练:并行Scalability的理论和实践限制
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability
摘要:
本文对深度神经网络(DNN)训练的可扩展分布式解决方案的主要瓶颈进行了理论分析和实践评估。所提出的结果表明,目前的最先进的方法,使用数据并行随机梯度下降(SGD),正在迅速变成一个巨大的通信约束问题。此外,我们提出了简单但固定的理论约束,防止DNN训练的有效扩展超过几十个节点。这导致DNN训练在大多数实际场景中的可扩展性较差。
英文摘要:
This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neuronal Networks (DNNs). The presented results show, that the current state of the art approach, using data-parallelized Stochastic Gradient Descent (SGD), is quickly turning into a vastly communication bound problem. In addition, we present simple but fixed theoretic constraints, preventing effective scaling of DNN training beyond only a few dozen nodes. This leads to poor scalability of DNN training in most practical scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。