arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.01075cs.DCcs.LGcs.PL
收藏

通过可移植的集体通信实现内存高效的阵列重新分配

Memory-efficient array redistribution through portable collective communication

DeepMind(深度思维);Google Research(谷歌研究院);EPFL(洛桑联邦理工学院)

Norman A. Rink, Adam Paszke, Dimitrios Vytiniotis, Georg Stefan Schmid

摘要:

现代大规模深度学习工作负载突出了跨多个设备并行执行的需求,以便将模型数据放入硬件加速器内存中。在这些设置中,在计算期间可能需要阵列重新分配,但如果没有高效地完成,也可能成为瓶颈。在本文中,我们解决了在SPMD计算中重新分配多维数组数据的问题,这是深度学习中最流行的并行形式。我们提出了一种类型导向的方法来合成数组重分布作为MPI风格的集体操作序列。我们正式证明,我们的合成重分布是内存效率和执行没有过多的数据传输。使用集体操作的SPMD计算的阵列重新分配也已经在XLA SPMD分区器的上下文中实现,XLA SPMD分区器是用于跨加速器系统分区程序的生产级工具。我们评估我们的方法对XLA的实现,发现我们的方法提供了一个几何平均加速$1.22\times$,最大加速高达$5.7\times$,同时提供可证明的内存保证,使我们的系统特别吸引人的大规模模型。

英文摘要:

Modern large-scale deep learning workloads highlight the need for parallel execution across many devices in order to fit model data into hardware accelerator memories. In these settings, array redistribution may be required during a computation, but can also become a bottleneck if not done efficiently. In this paper we address the problem of redistributing multi-dimensional array data in SPMD computations, the most prevalent form of parallelism in deep learning. We present a type-directed approach to synthesizing array redistributions as sequences of MPI-style collective operations. We prove formally that our synthesized redistributions are memory-efficient and perform no excessive data transfers. Array redistribution for SPMD computations using collective operations has also been implemented in the context of the XLA SPMD partitioner, a production-grade tool for partitioning programs across accelerator systems. We evaluate our approach against the XLA implementation and find that our approach delivers a geometric mean speedup of $1.22\times$, with maximum speedups as a high as $5.7\times$, while offering provable memory guarantees, making our system particularly appealing for large-scale models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑