arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.11150cs.LGcond-mat.mtrl-scics.DCphysics.comp-phstat.ML
收藏

科学逆问题的Exascale深度学习

Exascale Deep Learning for Scientific Inverse Problems

Oak Ridge National Laboratory(橡树岭国家实验室);NVIDIA(英伟达);Uber Technologies, Inc.(优步技术公司)

Nouamane Laanait, Joshua Romero, Junqi Yin, M. Todd Young, Sean Treichler, Vitalii Starchenko, Albina Borisevich, Alex Sergeev, Michael Matheson

摘要:

我们在同步分布式深度学习中引入了新的通信策略,包括分散的梯度约简编排和梯度张量的计算图感知分组。这些新技术在计算和通信之间产生了最佳的重叠,并在Summit超级计算机上实现了近线性缩放(0.93)的分布式训练,最多可达27,600个NVIDIA V100 GPU。我们展示了我们的梯度降低技术的背景下,训练一个完全卷积神经网络,以近似解决一个长期存在的科学逆问题,在材料成像。在0.5 PB的数据集大小上进行有效的分布式训练,产生能够对材料进行原子级精确重建的模型,并且在此过程中达到2.15(4)EFLOPS $_{16}$的峰值性能。

英文摘要:

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear scaling (0.93) of distributed training up to 27,600 NVIDIA V100 GPUs on the Summit Supercomputer. We demonstrate our gradient reduction techniques in the context of training a Fully Convolutional Neural Network to approximate the solution of a longstanding scientific inverse problem in materials imaging. The efficient distributed training on a dataset size of 0.5 PB, produces a model capable of an atomically-accurate reconstruction of materials, and in the process reaching a peak performance of 2.15(4) EFLOPS$_{16}$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑