arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.24326cs.DCcs.AIcs.NI
收藏

ScaleAcross Explorer:探索跨规模AI模型训练的通信优化

ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training

Harvard University(哈佛大学);Meta Platforms, Inc.(Meta平台公司)
Minghao Li, Alicia Golden, Samuel Hsia, Michael Kuchnik, Adi Gangidi, Xu Zhang, Ashmitha Jeevaraj Shetty, Zachary DeVito, Weiwei Chu, Dong He, Haoci Zhang, Yuch… 展开作者

Minghao Li, Alicia Golden, Samuel Hsia, Michael Kuchnik, Adi Gangidi, Xu Zhang, Ashmitha Jeevaraj Shetty, Zachary DeVito, Weiwei Chu, Dong He, Haoci Zhang, Yuchen Hao, Ruoming Pang, James Hongyi Zeng, Ying Zhang, Minlan Yu, Carole-Jean Wu

AI总结:

针对跨数据中心大规模AI模型训练(scale-across)的复杂设计空间,提出ScaleAcross Explorer优化器,通过联合优化并行放置、并行调度和网络层技术,实现高达64.62%的训练加速。

摘要:

大型语言模型训练的快速扩展需要将GPU资源分布在多个数据中心建筑和区域之间。我们将这种范式称为“scale-across”训练。随着基础设施的扩展,系统设计空间变得越来越复杂,涵盖了新的模型架构、硬件异构性和不断演变的通信模式。借鉴Meta的生产经验,我们强调了在跨多个拥有数十万GPU的数据中心部署训练作业的复杂性。为了加速对庞大设计空间的探索并实现前沿模型开发的高效训练,我们对三个关键设计维度进行了深入表征:并行放置、并行调度和网络层技术。然后,我们提出了ScaleAcross Explorer,这是一个考虑设计维度相互作用并整体优化跨规模训练的优化器。测试床实验和模拟表明,在广泛的设计点上,与生产配置相比,训练速度提升高达64.62%,与最先进的基线相比,训练速度提升高达37.59%。

英文摘要:

The rapid scaling of large language model training requires distributing GPU resources across multiple data center buildings and regions. We refer to such paradigm as "scale-across" training. As infrastructure expands, the system design space becomes increasingly intricate, encompassing new model architectures, hardware heterogeneity, and evolving communication patterns. Drawing from Meta's production experience, we highlight the complexities of deploying training jobs across a few data centers housing hundreds of thousands of GPUs. To accelerate exploration of the large design space and to enable efficient training for frontier model development, we conduct in-depth characterization of three key design dimensions: parallelism placement, parallelism scheduling, and network layer technologies. We then propose ScaleAcross Explorer, an optimizer that considers the interplay of design dimensions and holistically optimizes scale-across training. Testbed experiments and simulations demonstrate up to 64.62% training speedups over production configuration and up to 37.59% training speedups over the state-of-the-art baseline across a wide range of design points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑