arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.02483cs.LGcs.AIcs.CL
收藏

Litespark技术报告:高吞吐量、高能效的大语言模型训练框架

Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework

Mindbeam AI

Nii Osae Osae Dade, Moinul Hossain Rahat

AI总结:

针对大语言模型训练耗时长、能耗高的问题,研究提出Litespark预训练框架,通过优化Transformer注意力与MLP层提升MFU,在多节点H200集群上实现2-6倍吞吐量提升与55%-83%能耗降低,且适配多类架构与训练后阶段。

摘要:

大语言模型(LLM)的训练长期受训练时间长、能耗巨大的问题困扰,当代模型需要数月的计算时长与数吉瓦时的电力消耗。针对这些挑战,我们提出Litespark这一新型预训练框架,通过对Transformer注意力层与MLP层的定向优化解决上述低效问题。该方法结合架构改进与算法增强,在保持与标准Transformer实现兼容性的同时,最大化模型浮点运算利用率(MFU)。我们使用SlimPajama-627B数据集,在3B和30B参数的Llama模型上开展了全面基准测试,结果显示性能提升显著:在多节点H200 GPU集群上,训练吞吐量提升2倍至6倍,能耗降低55%-83%。这些优化不依赖特定模型与硬件,可广泛适用于各类Transformer架构,还能拓展至监督微调、直接偏好优化等训练后阶段。

英文摘要:

Training Large Language Models (LLMs) is plagued by long training times and massive energy consumption, with modern models requiring months of computation and gigawatt-hours of electricity. In light of these challenges,we introduce Litespark, a novel pre-training framework that addresses these inefficiencies through targeted optimizations to transformer attention and MLP layers. Our approach combines architectural improvements with algorithmic enhancements to maximize Model FLOPs Utilization (MFU) while maintaining compatibility with standard transformer implementations. Comprehensive benchmarking on 3B and 30B parameter Llama models using the SlimPajama-627B dataset demonstrates substantial performance gains: 2x-6x training throughput improvement and $55\%-83$% energy consumption reduction across multi-node H200 GPU clusters. These optimizations are model- and hardware-agnostic, enabling broad applicability across transformer architectures and extending to post-training phases including supervised fine-tuning and direct preference optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑