arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.18093cs.DCcs.LG
收藏

Pipette:面向真实世界集群的自动细粒度大语言模型训练配置器

Pipette: Automatic Fine-grained Large Language Model Training Configurator for Real-World Clusters

Seoul National University(首尔大学);Samsung Electronics(三星电子)

Jinkyu Yim, Jaeyong Song, Yerim Choi, Jaebeen Lee, Jaewon Jung, Hongsun Jang, Jinho Lee

AI总结:

针对真实集群中3D并行LLM训练配置忽视互连异构性和内存约束的问题,提出Pipette,通过改进性能模型、内存估计和细粒度GPU分配,实现满足内存约束的更快配置并显著加速。

摘要:

训练大型语言模型(LLM)因巨大的计算和内存容量需求而众所周知地具有挑战性。为解决这些问题,通常使用带有3D并行性的GPU集群,该并行性沿数据批次、流水线阶段和层内张量维度对模型进行切分。然而,使用3D并行性带来了额外的挑战,即需要找到每个维度上的最优切分数量并将切分后的模型映射到GPU上。已有若干先前研究尝试自动寻找最优配置,但其中许多缺乏若干重要方面。例如,互连速度的异构性常被忽略。虽然互连的峰值带宽通常被设为相等,但在真实世界的集群中,每条链路实际达到的带宽各不相同。结合未适当考虑通信的关键路径建模,它们容易陷入次优配置。此外,它们往往未能考虑每GPU的内存需求,常推荐无法执行的解决方案。为应对这些挑战,我们提出Pipette,一种面向真实世界集群的自动细粒度LLM训练配置器。通过设计更好的性能模型以及内存估计器和细粒度个体GPU分配,Pipette实现了满足内存约束的更快配置。我们在大型集群上评估了Pipette,表明其相对于先前技术提供了显著的加速。Pipette的实现可在https://github.com/yimjinkyu1/date2024_pipette获取。

英文摘要:

Training large language models (LLMs) is known to be challenging because of the huge computational and memory capacity requirements. To address these issues, it is common to use a cluster of GPUs with 3D parallelism, which splits a model along the data batch, pipeline stage, and intra-layer tensor dimensions. However, the use of 3D parallelism produces the additional challenge of finding the optimal number of ways on each dimension and mapping the split models onto the GPUs. Several previous studies have attempted to automatically find the optimal configuration, but many of these lacked several important aspects. For instance, the heterogeneous nature of the interconnect speeds is often ignored. While the peak bandwidths for the interconnects are usually made equal, the actual attained bandwidth varies per link in real-world clusters. Combined with the critical path modeling that does not properly consider the communication, they easily fall into sub-optimal configurations. In addition, they often fail to consider the memory requirement per GPU, often recommending solutions that could not be executed. To address these challenges, we propose Pipette, which is an automatic fine-grained LLM training configurator for real-world clusters. By devising better performance models along with the memory estimator and fine-grained individual GPU assignment, Pipette achieves faster configurations that satisfy the memory constraints. We evaluated Pipette on large clusters to show that it provides a significant speedup over the prior art. The implementation of Pipette is available at https://github.com/yimjinkyu1/date2024_pipette.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑