arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.02868cs.LGcs.DC
收藏

Angel-PTM:腾讯一个可扩展、经济的大规模预培训系统

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

Peking University(北京大学);Tencent Inc.(腾讯公司);Carnegie Mellon University(卡内基梅隆大学)

Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, Bin Cui

摘要:

近年来,大规模预训练模型取得了前所未有的成就,特别是Transformer模型。腾讯公司的许多产品和服务,例如微信,QQ和腾讯广告,已经被选择来获得预先训练的模型的力量。在这项工作中,我们介绍了Angel-PTM,这是一个高效的深度学习系统,旨在预训练和微调Transformer模型。Angel-PTM可以有效地训练具有层次存储的超大规模模型。Angel-PTM的关键设计是通过页面抽象和协调计算、数据移动和通信的统一调度方法进行细粒度存储管理。此外,Angel-PTM支持SSD存储的极端模型扩展,并实现无锁更新机制以解决SSD I/O带宽瓶颈。实验结果表明,Angel-PTM在最大模型规模方面优于现有系统高达114.8%,在训练吞吐量方面则高达88.9%。此外,在GPT 3 - 175 B和T5-MoE-1.2T模型上使用数百个GPU的实验验证了Angel-PTM的强大可扩展性。

英文摘要:

Recent years have witnessed the unprecedented achievements of large-scale pre-trained models, especially the Transformer models. Many products and services in Tencent Inc., such as WeChat, QQ, and Tencent Advertisement, have been opted in to gain the power of pre-trained models. In this work, we present Angel-PTM, a productive deep learning system designed for pre-training and fine-tuning Transformer models. Angel-PTM can train extremely large-scale models with hierarchical memory efficiently. The key designs of Angel-PTM are the fine-grained memory management via the Page abstraction and a unified scheduling method that coordinate the computations, data movements, and communications. Furthermore, Angel-PTM supports extreme model scaling with SSD storage and implements the lock-free updating mechanism to address the SSD I/O bandwidth bottlenecks. Experimental results demonstrate that Angel-PTM outperforms existing systems by up to 114.8% in terms of maximum model scale as well as up to 88.9% in terms of training throughput. Additionally, experiments on GPT3-175B and T5-MoE-1.2T models utilizing hundreds of GPUs verify the strong scalability of Angel-PTM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑