在MXNET DAG中有效嵌入MPI集合以扩展深度学习
Efficient Embedding of MPI Collectives in MXNET DAGs for scaling Deep Learning
摘要：
高性能计算基础设施（如GPU和CPU集群）的可用性推动了分布式学习系统的发展。深度学习框架将神经网络表示为DAG，并在GPU等计算资源上执行这些DAG。在本文中，我们提出了有效的设计嵌入MPI集体操作到数据并行DAG。不正确的设计很容易导致死锁或程序崩溃。特别是，我们展示了三种设计：漏斗，并发通信和依赖链使用MPI集体与DAG。这些设计通过允许与其他任务的并发执行来自动地实现计算与通信的重叠。我们直接将这些设计实现到MXNET的KVStore API中。这使我们能够直接利用其余的基础设施。使用ImageNet和CIFAR数据集，我们展示了我们设计的潜力。特别是，我们的设计可扩展到256个GPU，ImageNet 1K数据集的历元时间低至50秒。
英文摘要：
Availability of high performance computing infrastructures such as clusters of GPUs and CPUs have fueled the growth of distributed learning systems. Deep Learning frameworks express neural nets as DAGs and execute these DAGs on computation resources such as GPUs. In this paper, we propose efficient designs of embedding MPI collective operations into data parallel DAGs. Incorrect designs can easily lead to deadlocks or program crashes. In particular, we demonstrate three designs: Funneled, Concurrent communication and Dependency chaining of using MPI collectives with DAGs. These designs automatically enable overlap of computation with communication by allowing for concurrent execution with the other tasks. We directly implement these designs into the KVStore API of the MXNET. This allows us to directly leverage the rest of the infrastructure. Using ImageNet and CIFAR data sets, we show the potential of our designs. In particular, our designs scale to 256 GPUs with as low as 50 seconds of epoch times for ImageNet 1K datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
