arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.13143cs.LGcs.ARcs.DC
收藏

集成硬件架构与设备放置搜索

Integrated Hardware Architecture and Device Placement Search

Georgia Institute of Technology(佐治亚理工学院);Microsoft Research(微软研究院)

Irene Wang, Jakub Tarnawski, Amar Phanishayee, Divya Mahajan

AI总结:

本文首次提出协同优化硬件架构与设备放置策略的算法,利用ILP和动态规划,在TPUv4和Spotlight之上提升大语言模型训练吞吐量。

摘要:

深度学习训练中的分布式执行涉及硬件加速器架构与设备放置策略之间的动态交互。这是首个通过新颖算法探索确定最优架构与设备放置策略协同优化的研究工作,旨在改善计算资源、内存使用和数据分布的平衡。我们的架构搜索利用张量和向量单元,确定其数量与维度,以及片上与片外内存配置。它还确定微批量大小,并决定是重新计算还是暂存激活,以平衡训练的内存占用与存储大小。对于每种探索的架构配置,我们使用整数线性规划(ILP)来寻找在加速器上执行算子的最优调度。ILP结果随后与动态规划解决方案集成,以识别最有效的设备放置策略,结合跨多个加速器的数据、流水线和张量模型并行。与最先进的TPUv4和Spotlight加速器搜索框架相比,我们的方法在大语言模型上实现了更高的吞吐量。PHAZE的完整源代码可在https://github.com/msr-fiddle/phaze获取。

英文摘要:

Distributed execution of deep learning training involves a dynamic interplay between hardware accelerator architecture and device placement strategy. This is the first work to explore the co-optimization of determining the optimal architecture and device placement strategy through novel algorithms, improving the balance of computational resources, memory usage, and data distribution. Our architecture search leverages tensor and vector units, determining their quantity and dimensionality, and on-chip and off-chip memory configurations. It also determines the microbatch size and decides whether to recompute or stash activations, balancing the memory footprint of training and storage size. For each explored architecture configuration, we use an Integer Linear Program (ILP) to find the optimal schedule for executing operators on the accelerator. The ILP results then integrate with a dynamic programming solution to identify the most effective device placement strategy, combining data, pipeline, and tensor model parallelism across multiple accelerators. Our approach achieves higher throughput on large language models compared to the state-of-the-art TPUv4 and the Spotlight accelerator search framework. The entire source code of PHAZE is available at https://github.com/msr-fiddle/phaze.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑