arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02108cs.CL
收藏

突破内存瓶颈:支持百万级上下文的高效训练系统

Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts

Key Laboratory of Multimedia Trusted Perception and Efficient Computing Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Xiamen University(厦门大学);Peking University(北京大学);University of Illinois Urbana-Champaign(伊利诺伊大学香槟分校);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室)

Wenhao Li, Daohai Yu, Gen Luo, Yuxin Zhang, Fei Chao, Rongrong Ji, Yifan Wu, Jiaxin Liu, Ziyang Gong, Zimu Liao

AI总结:

OOMB通过高效内存管理实现百万级上下文LLM训练,显著降低内存开销,提升训练效率。

摘要:

训练大型语言模型(LLMs)于长上下文严重受限于GPU内存开销,而非训练时间。主要原因是激活值,其内存足迹与序列长度成线性关系。我们引入OOMB,一种高度内存高效的训练系统,直接面对这一瓶颈。我们的方法采用分块递归训练框架,结合即时激活值重新计算,保持激活值内存足迹恒定(O(1)),并将主要瓶颈转移到增长的KV缓存。为管理KV缓存,OOMB集成了协同优化:为KV缓存及其梯度的分页内存管理以消除碎片,异步CPU卸载以隐藏数据传输延迟,以及基于页面的稀疏注意力以减少计算复杂度和通信开销。这些技术的协同效应带来了极高的效率。我们的实证结果表明,对于每增加10K个上下文token,端到端训练内存开销仅增加10MB(Qwen2.5-7B)。这使得可以在单个H200 GPU上训练具有4M token上下文的Qwen2.5-7B,这通常需要使用上下文并行的大型集群。这项工作在长上下文LLM训练的资源效率方面取得了重大进展。源代码可在https://github.com/wenhaoli-xmu/OOMB获取。

英文摘要:

Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time. The primary culprits are the activations, whose memory footprints scale linearly with sequence length. We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier. Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint (O(1)) and shifts the primary bottleneck to the growing KV cache. To manage the KV cache, OOMB integrates a suite of synergistic optimizations: a paged memory manager for both the KV cache and its gradients to eliminate fragmentation, asynchronous CPU offloading to hide data transfer latency, and page-level sparse attention to reduce both computational complexity and communication overhead. The synergy of these techniques yields exceptional efficiency. Our empirical results show that for every additional 10K tokens of context, the end-to-end training memory overhead increases by a mere 10MB for Qwen2.5-7B. This allows training Qwen2.5-7B with a 4M-token context on a single H200 GPU, a feat that would otherwise require a large cluster using context parallelism. This work represents a substantial advance in resource efficiency for long-context LLM training. The source code is available at https://github.com/wenhaoli-xmu/OOMB.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑