在设备放置中利用高效图遍历顺序加速模型并行训练
Accelerate Model Parallel Training by Using Efficient Graph Traversal Order in Device Placement
AI总结：
本文研究模型并行训练中图遍历顺序对设备放置及训练时间的影响，通过实证分析发现最佳顺序取决于网络类型与计算图特征，并给出选型建议。
摘要：
现代神经网络需要长时间训练，才能在大规模数据集上达到良好性能。加速训练的一种常用方法是模型并行化，即将大型神经网络拆分到多个设备上。然而，同一神经网络的不同设备放置会导致不同的训练时间。现有大多数设备放置解决方案将该问题视为序列决策，通过遍历神经网络图并将其神经元分配到不同设备来完成。本工作研究图遍历顺序对设备放置的影响。具体而言，我们通过实证研究不同图遍历顺序如何导致不同的设备放置，进而影响训练执行时间。实验结果表明，最佳图遍历顺序取决于神经网络的类型及其计算图特征。在本工作中，我们还针对不同神经网络系列提供了在设备放置中选择图遍历顺序的建议，以改善模型并行化中的训练时间。
英文摘要：
Modern neural networks require long training to reach decent performance on massive datasets. One common approach to speed up training is model parallelization, where large neural networks are split across multiple devices. However, different device placements of the same neural network lead to different training times. Most of the existing device placement solutions treat the problem as sequential decision-making by traversing neural network graphs and assigning their neurons to different devices. This work studies the impact of graph traversal order on device placement. In particular, we empirically study how different graph traversal order leads to different device placement, which in turn affects the training execution time. Our experiment results show that the best graph traversal order depends on the type of neural networks and their computation graphs features. In this work, we also provide recommendations on choosing graph traversal order in device placement for various neural network families to improve the training time in model parallelization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
