AI民主化:基于GPU的超级计算机上的开源可扩展LLM训练
Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers
AI总结:
提出基于开源框架AxoNN的四维混合并行算法及优化,实现GPT模型在多超级计算机上的高扩展训练,同时通过实验揭示大模型灾难性记忆风险并提出预防方法,还展示了4050亿参数LLM的微调能力。
摘要:
训练和微调具有数千亿到数万亿参数的大型语言模型(LLMs)需要数万个GPU和高度可扩展的软件栈。在这项工作中,我们提出了一种新颖的四维混合并行算法,该算法在名为AxoNN的高度可扩展、可移植的开源框架中实现。我们描述了AxoNN中的几项性能优化,包括提高矩阵乘法内核性能、将非阻塞集合操作与计算重叠,以及通过性能建模选择性能最优配置。这些优化在Perlmutter(620.1 Petaflop/s)、Frontier(1.381 Exaflop/s)和Alps(1.423 Exaflop/s)上实现了GPT风格Transformer模型训练的前所未有的扩展性和峰值浮点运算速度(bf16)。虽然LLMs的能力随可训练参数数量的增加而提升,但训练数据记忆导致的隐私和版权风险也随之增加,这可能在推理时导致敏感或私人信息泄露。我们通过探索“灾难性记忆”的实验强调了这种规模的副作用——模型足够大时可在单次传递中记忆训练数据,并提出了一种预防方法。作为本研究的一部分,我们展示了使用AxoNN在Frontier上微调4050亿参数LLM的能力。
英文摘要:
Training and fine-tuning large language models (LLMs) with hundreds of billions to trillions of parameters requires tens of thousands of GPUs, and a highly scalable software stack. In this work, we present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN. We describe several performance optimizations in AxoNN to improve matrix multiply kernel performance, overlap non-blocking collectives with computation, and performance modeling to choose performance optimal configurations. These have resulted in unprecedented scaling and peak flop/s (bf16) for training of GPT-style transformer models on Perlmutter (620.1 Petaflop/s), Frontier (1.381 Exaflop/s) and Alps (1.423 Exaflop/s). While the abilities of LLMs improve with the number of trainable parameters, so do privacy and copyright risks caused by memorization of training data, which can cause disclosure of sensitive or private information at inference time. We highlight this side effect of scale through experiments that explore "catastrophic memorization", where models are sufficiently large to memorize training data in a single pass, and present an approach to prevent it. As part of this study, we demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。