arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.20437cs.LGcs.AI
收藏

GaLore 2:通过梯度低秩投影实现大规模 LLM 预训练

GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection

FAIR at Meta AI(Meta AI 的 FAIR 研究院);PyTorch

DiJia Su, Andrew Gu, Jane Xu, Yuandong Tian, Jiawei Zhao

AI总结:

本文提出高效可扩展的 GaLore 2 框架,通过解决 SVD 计算开销和并行化策略集成等挑战,并使用多达 5000 亿 token 从零预训练 Llama 7B,展现了其在真实大规模 LLM 预训练中的潜力。

摘要:

大型语言模型(LLMs)彻底改变了自然语言理解与生成,但在训练过程中面临显著的内存瓶颈。GaLore(Gradient Low-Rank Projection,梯度低秩投影)通过利用权重梯度的内在低秩结构来解决此问题,在不牺牲性能的情况下实现了大量内存节省。近期的研究从低比特量化和高阶张量结构等方面进一步扩展了 GaLore。然而,GaLore 仍存在一些挑战,如用于子空间更新的 SVD 计算开销,以及与最先进训练并行化策略(如 FSDP)的集成。本文提出 GaLore 2,这是一个高效且可扩展的 GaLore 框架,解决了上述挑战并整合了近期进展。此外,我们通过使用多达 5000 亿个训练 token 从零开始预训练 Llama 7B,展示了 GaLore 2 的可扩展性,突出了其对真实 LLM 预训练场景的潜在影响。

英文摘要:

Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Projection, addresses this issue by leveraging the inherent low-rank structure of weight gradients, enabling substantial memory savings without sacrificing performance. Recent works further extend GaLore from various aspects, including low-bit quantization and higher-order tensor structures. However, there are several remaining challenges for GaLore, such as the computational overhead of SVD for subspace updates and the integration with state-of-the-art training parallelization strategies (e.g., FSDP). In this paper, we present GaLore 2, an efficient and scalable GaLore framework that addresses these challenges and incorporates recent advancements. In addition, we demonstrate the scalability of GaLore 2 by pre-training Llama 7B from scratch using up to 500 billion training tokens, highlighting its potential impact on real LLM pre-training scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑