在Cori超级计算机的512个节点上扩展GRPC Tensorflow
Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer
摘要:
我们探索了标准分布式Tensorflow与GRPC原语在Cori超级计算机的多达512个Intel Xeon Phi(KNL)节点上的扩展,同步随机梯度下降(SGD),并确定了在更高节点数下扩展效率低下的原因。据我们所知,这是首次在HPC超级计算机上使用同步SGD大规模探索分布式GRPC Tensorflow可扩展性。我们研究了两个卷积神经网络的缩放- ResNet-50,一个用于分类的最先进的深度网络,大约有2550万个参数,以及HEP-CNN,一个用于常见科学用途的参数少于100万个的浅层拓扑结构。对于ResNet-50,我们使用32个参数服务器(PS任务)在多达128个工作者上实现了>80%的扩展效率,而使用64个PS任务的512个工作者的扩展效率急剧下降到23%。我们对效率下降的分析指出,由于三个因素的综合影响,网络带宽利用率较低。(a)异构分布式并行算法使用PS任务作为集中式服务器进行梯度平均,对于利用互连带宽是次优的。(b)PS任务之间的负载不平衡阻碍了它们的有效扩展。(c)底层通信原语GRPC目前在Cori高速互连上效率低下。HEP-CNN需要较少的互连带宽,并且对于多达256个节点仅具有1个PS任务显示>80%的弱缩放效率。我们的发现也适用于其他深度学习网络。拥有数百万参数的大型网络偶然发现了这里讨论的问题。像HEP-CNN这样参数数量相对较少的浅网络即使使用单个参数服务器也可以有效地享受弱缩放。
英文摘要:
We explore scaling of the standard distributed Tensorflow with GRPC primitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer with synchronous stochastic gradient descent (SGD), and identify causes of scaling inefficiency at higher node counts. To our knowledge, this is the first exploration of distributed GRPC Tensorflow scalability on a HPC supercomputer at such large scale with synchronous SGD. We studied scaling of two convolution neural networks - ResNet-50, a state-of-the-art deep network for classification with roughly 25.5 million parameters, and HEP-CNN, a shallow topology with less than 1 million parameters for common scientific usages. For ResNet-50, we achieve >80% scaling efficiency on up to 128 workers, using 32 parameter servers (PS tasks) with a steep decline down to 23% for 512 workers using 64 PS tasks. Our analysis of the efficiency drop points to low network bandwidth utilization due to combined effect of three factors. (a) Heterogeneous distributed parallelization algorithm which uses PS tasks as centralized servers for gradient averaging is suboptimal for utilizing interconnect bandwidth. (b) Load imbalance among PS tasks hinders their efficient scaling. (c) Underlying communication primitive GRPC is currently inefficient on Cori high-speed interconnect. The HEP-CNN demands less interconnect bandwidth, and shows >80% weak scaling efficiency for up to 256 nodes with only 1 PS task. Our findings are applicable to other deep learning networks. Big networks with millions of parameters stumble upon the issues discussed here. Shallower networks like HEP-CNN with relatively lower number of parameters can efficiently enjoy weak scaling even with a single parameter server.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。