使用分布式GPU预训练大型语言模型：一种内存高效的分散式范式
Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm
AI总结：
提出SPES框架，通过分散式训练MoE LLM的子集专家降低内存需求，结合专家合并预热策略，在16个48GB GPU上训练2B参数模型，性能媲美集中式训练。
摘要：
预训练大型语言模型（LLMs）通常需要配备数千个高内存GPU（如H100/A100）的集中式集群。最近的分散式训练方法通过采用联邦优化来减少通信开销；然而，它们仍然需要在每个节点上训练整个模型，因此仍受限于GPU内存限制。在这项工作中，我们提出了SPES（稀疏专家同步），一种用于预训练混合专家（MoE）LLMs的内存高效分散式框架。SPES在每个节点上仅训练一部分专家，大幅降低了内存占用。每个节点更新其本地专家，并定期与其他节点同步，消除了全参数传输，同时确保高效的知识共享。为了缓解稀疏专家更新下每个专家数据利用率有限的问题，我们引入了一种专家合并预热策略，即在训练早期让专家交换知识，以快速建立基础能力。通过SPES，我们使用16个独立的48GB GPU通过互联网连接训练了一个2B参数的MoE LLM，在相似计算预算下取得了与集中式训练LLM相竞争的性能。我们进一步展示了可扩展性，从头开始训练了一个7B模型，并从密集检查点升级了一个9B模型，两者均匹配先前的集中式基线。我们的代码可在该https URL获取。
英文摘要：
Pretraining large language models (LLMs) typically requires centralized clusters with thousands of high-memory GPUs (e.g., H100/A100). Recent decentralized training methods reduce communication overhead by employing federated optimization; however, they still need to train the entire model on each node, remaining constrained by GPU memory limitations. In this work, we propose SParse Expert Synchronization (SPES), a memory-efficient decentralized framework for pretraining mixture-of-experts (MoE) LLMs. SPES trains only a subset of experts per node, substantially lowering the memory footprint. Each node updates its local experts and periodically synchronizes with other nodes, eliminating full-parameter transmission while ensuring efficient knowledge sharing. To mitigate limited per-expert data utilization under sparse expert updates, we introduce an expert-merging warm-up strategy, where experts exchange knowledge early in training, to rapidly establish foundational capabilities. With SPES, we train a 2B-parameter MoE LLM using 16 standalone 48GB GPUs over internet connections, which achieves competitive performance with centrally trained LLMs under similar computational budgets. We further demonstrate scalability by training a 7B model from scratch and a 9B model upcycled from a dense checkpoint, both of which match prior centralized baselines. Our code is available at https://github.com/zjr2000/SPES.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
