面向超大规模深度学习的自动图划分
Automatic Graph Partitioning for Very Large-scale Deep Learning
摘要:
这项工作提出了RaNNC(快速神经网络连接器)作为自动混合并行的中间件。在最近的深度学习研究中,以T5和GPT-3为例,神经网络模型的规模持续增长。由于这些模型不适合加速器设备的存储器,因此需要通过模型并行技术对它们进行划分。此外,为了加速对巨大训练数据的训练,我们需要模型和数据并行的组合,即,混合并行给定PyTorch的模型描述,而没有任何模型并行性的规范,RaNNC自动将模型划分为一组子组件,以便(1)每个子组件适合设备内存,以及(2)通过平衡子组件的计算时间来实现流水线并行的高训练吞吐量。在我们的实验中,我们比较了RaNNC与两个流行的框架,Megatron-LM(混合并行)和GPipe(最初为模型并行提出,但也存在允许混合并行的版本),用于训练参数数量越来越多的模型。在扩大的BERT模型的预训练中,RaNNC成功地训练了比Megatron-LM大五倍的模型,并且RaNNC的训练吞吐量与Megatron-LM在预训练相同模型时相当。在我们尝试的所有设置中,RaNNC在扩大的BERT模型预训练(具有混合并行性的GPipe)和扩大的ResNet模型(具有模型并行的GPipe)上都取得了比GPipe更好的训练吞吐量。这些结果是显著的,因为RaNNC自动划分模型而不修改其描述; Megatron-LM和GPipe需要用户手动重写模型的描述。
英文摘要:
This work proposes RaNNC (Rapid Neural Network Connector) as middleware for automatic hybrid parallelism. In recent deep learning research, as exemplified by T5 and GPT-3, the size of neural network models continues to grow. Since such models do not fit into the memory of accelerator devices, they need to be partitioned by model parallelism techniques. Moreover, to accelerate training for huge training data, we need a combination of model and data parallelisms, i.e., hybrid parallelism. Given a model description for PyTorch without any specification for model parallelism, RaNNC automatically partitions the model into a set of subcomponents so that (1) each subcomponent fits a device memory and (2) a high training throughput for pipeline parallelism is achieved by balancing the computation times of the subcomponents. In our experiments, we compared RaNNC with two popular frameworks, Megatron-LM (hybrid parallelism) and GPipe (originally proposed for model parallelism, but a version allowing hybrid parallelism also exists), for training models with increasingly greater numbers of parameters. In the pre-training of enlarged BERT models, RaNNC successfully trained models five times larger than those Megatron-LM could, and RaNNC's training throughputs were comparable to Megatron-LM's when pre-training the same models. RaNNC also achieved better training throughputs than GPipe on both the enlarged BERT model pre-training (GPipe with hybrid parallelism) and the enlarged ResNet models (GPipe with model parallelism) in all of the settings we tried. These results are remarkable, since RaNNC automatically partitions models without any modification to their descriptions; Megatron-LM and GPipe require users to manually rewrite the models' descriptions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。