ZeRO-Offload:使十亿规模的模型训练民主化
ZeRO-Offload: Democratizing Billion-Scale Model Training
摘要:
大规模模型训练一直是少数几个需要复杂模型重构和访问昂贵GPU集群的游戏场地。ZeRO-Offload通过使几乎所有人都可以使用大型模型训练来改变大型模型训练的格局。它可以在单个GPU上训练具有超过130亿个参数的模型,与PyTorch等流行框架相比,大小增加了10倍,并且不需要数据科学家对模型进行任何更改或牺牲计算效率。ZeRO-Offload通过将数据和计算卸载到CPU来实现大型模型训练。为了保持计算效率,它旨在最大限度地减少数据移动到GPU/从GPU,并减少CPU计算时间,同时最大限度地节省GPU上的内存。因此,ZeRO-Offload可以在单个NVIDIA V100 GPU上为10 B参数模型实现40 TFlops/GPU,而单独使用PyTorch为1.4B参数模型实现30 TF,这是可以在不耗尽内存的情况下进行训练的最大值。ZeRO-Offload还可以在多GPU上扩展,在多达128个GPU上提供接近线性的加速。此外,它可以与模型并行一起工作,在单个DGX-2框上训练具有超过700亿个参数的模型,与单独使用模型并行相比,模型大小增加了4.5倍。通过将计算和内存效率与易用性相结合,ZeRO-Offload使大规模模型训练民主化,即使是数据科学家也可以访问单个GPU。
英文摘要:
Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。