arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.08984cs.DCcs.AI
收藏

SuperScaler:通过统一抽象支持灵活的DNN并行化

SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction

University of Science and Technology of China(中国科学技术大学);Microsoft Research(微软研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Shanghai Jiao Tong University(上海交通大学)

Zhiqi Lin, Youshan Miao, Guodong Liu, Xiaoxiang Shi, Quanlu Zhang, Fan Yang, Saeed Maleki, Yi Zhu, Xu Cao, Cheng Li, Mao Yang, Lintao Zhang, Lidong Zhou

摘要:

随着模型规模的不断增长,深度神经网络(DNN)越来越多地在大规模GPU加速器上进行训练,这需要一个适当的并行化计划,将DNN模型转换为细粒度的任务,然后将它们调度到GPU上执行。由于大的搜索空间,当代的并行化计划生成器通常依赖于耦合转换和调度的经验规则,并且在探索产生更好的存储器使用和计算效率的更灵活的调度方面不足。这种紧张关系可能会因其结构和模型规模日益复杂的新兴模型而加剧。SuperScaler是一个便于设计和生成高度灵活的并行化计划的系统。它明确地将计划的设计和生成分成三个连续的阶段:模型转换、时空调度和数据依赖保持。这种原则性的方法将多个看似相互交织的因素结合起来,并实现了高度灵活的并行化计划的组成。因此,SuperScaler不仅可以生成经验并行化计划,还可以构建新的计划,与DeepSpeed,Megatron和Alpa等最先进的解决方案相比,可以实现高达3.5倍的加速,适用于Swin-Transformer和AlphaFold 2等新兴DNN模型以及GPT-3等优化模型。

英文摘要:

With the growing model size, deep neural networks (DNN) are increasingly trained over massive GPU accelerators, which demands a proper parallelization plan that transforms a DNN model into fine-grained tasks and then schedules them to GPUs for execution. Due to the large search space, the contemporary parallelization plan generators often rely on empirical rules that couple transformation and scheduling, and fall short in exploring more flexible schedules that yield better memory usage and compute efficiency. This tension can be exacerbated by the emerging models with increasing complexity in their structure and model size. SuperScaler is a system that facilitates the design and generation of highly flexible parallelization plans. It formulates the plan design and generation into three sequential phases explicitly: model transformation, space-time scheduling, and data dependency preserving. Such a principled approach decouples multiple seemingly intertwined factors and enables the composition of highly flexible parallelization plans. As a result, SuperScaler can not only generate empirical parallelization plans, but also construct new plans that achieve up to 3.5X speedup compared to state-of-the-art solutions like DeepSpeed, Megatron and Alpa, for emerging DNN models like Swin-Transformer and AlphaFold2, as well as well-optimized models like GPT-3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑