arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.16121cs.DC
收藏

Oases:通过重叠和自动化的张量模型并行在商用服务器上实现高效的大规模模型训练

Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism

Shengwei Li, Zhiquan Lai, Dongsheng Li, Yanqi Hao, Weijie Liu, Keshi Ge, Xiaoge Deng, Kai Lu

AI总结:

本文提出Oases,一种面向商用服务器的自动化张量模型并行方法,通过细粒度调度最大化重叠通信与计算,并设计规划器优化参数划分,在多种模型和集群上实现显著加速。

摘要:

深度学习正经历大规模模型的兴起。训练大规模模型成本高昂,促使研究人员在更多研究者可访问的商用服务器上训练大规模模型。海量的参数使得必须使用模型并行训练方法。现有研究侧重于使用流水线模型并行进行训练。然而,当模型规模持续增大时,张量模型并行(TMP)是不可避免的,其中频繁的数据依赖通信和计算操作显著降低了训练效率。在本文中,我们提出了Oases,一种具有重叠通信的自动化TMP方法,以加速商用服务器上的大规模模型训练。Oases提出了一种细粒度的训练操作调度,以最大化具有数据依赖的重叠通信和计算。此外,我们设计了Oases规划器,它搜索TMP的最佳模型参数划分策略以实现进一步加速。与现有方法不同,Oases规划器专为建模重叠通信-计算操作的成本而定制。我们在各种模型设置和两个商用集群上评估了Oases,并将Oases与四种最先进的实现进行了比较。实验结果表明,Oases相比最快的基线实现了1.01-1.48倍的加速,相比Megatron实现了高达1.95倍的加速。

英文摘要:

Deep learning is experiencing a rise in large-scale models. Training large-scale models is costly, prompting researchers to train large-scale models on commodity servers that more researchers can access. The massive number of parameters necessitates the use of model parallelism training methods. Existing studies focus on training with pipeline model parallelism. However, the tensor model parallelism (TMP) is inevitable when the model size keeps increasing, where frequent data-dependent communication and computation operations significantly reduce the training efficiency. In this paper, we present Oases, an automated TMP method with overlapped communication to accelerate large-scale model training on commodity servers. Oases proposes a fine-grained training operation schedule to maximize overlapping communication and computation that have data dependence. Additionally, we design the Oases planner that searches for the best model parameter partition strategy of TMP to achieve further accelerations. Unlike existing methods, Oases planner is tailored to model the cost of overlapped communication-computation operations. We evaluate Oases on various model settings and two commodity clusters, and compare Oases to four state-of-the-art implementations. Experimental results show that Oases achieves speedups of 1.01--1.48\(\times\) over the fastest baseline, and speedups of up to 1.95\(\times\) over Megatron.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑