arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.11277cs.DCcs.AIcs.LGcs.PF
收藏

PyTorch FSDP:扩展全分片数据并行的经验

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Meta AI
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, … 展开作者

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, Shen Li

AI总结:

本文介绍了PyTorch FSDP,一种大模型训练的工业级解决方案,通过与PyTorch核心组件协同设计提供高效且非侵入式的训练体验,并在支持更大模型的同时实现近线性扩展。

摘要:

人们普遍认为,大模型有潜力在广泛的领域中提供卓越的性能。尽管机器学习系统研究领域取得了显著进展,使得大模型的开发和探索成为可能,但这些能力仍然局限于少数高级用户和行业领导者,这为更广泛的社区访问和利用这些技术造成了隐性的技术壁垒。在本文中,我们介绍了 PyTorch Fully Sharded Data Parallel (FSDP) 作为大模型训练的工业级解决方案。FSDP 与多个 PyTorch 核心组件进行了紧密的协同设计,包括 Tensor 实现、dispatcher 系统和 CUDA 内存缓存分配器,以提供非侵入式的用户体验和高训练效率。此外,FSDP 原生集成了多种技术和设置,以在各种硬件配置上优化资源利用率。实验结果表明,FSDP 能够实现与 Distributed Data Parallel 相当的性能,同时在 TFLOPS 方面具有近线性的可扩展性,并支持显著更大的模型。

英文摘要:

It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of machine learning systems research, which has enabled the development and exploration of large models, such abilities remain confined to a small group of advanced users and industry leaders, resulting in an implicit technical barrier for the wider community to access and leverage these technologies. In this paper, we introduce PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution for large model training. FSDP has been closely co-designed with several key PyTorch core components including Tensor implementation, dispatcher system, and CUDA memory caching allocator, to provide non-intrusive user experiences and high training efficiency. Additionally, FSDP natively incorporates a range of techniques and settings to optimize resource utilization across a variety of hardware configurations. The experimental results demonstrate that FSDP is capable of achieving comparable performance to Distributed Data Parallel while providing support for significantly larger models with near-linear scalability in terms of TFLOPS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑