谱紧凑训练:通过永久截断SVD和Stiefel QR回退预训练大语言模型
Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction
AI总结:
本文提出SCT方法,通过永久截断SVD和Stiefel QR回退减少内存占用,实现大语言模型的高效预训练,实验显示内存占用降低达199倍,提升训练效率。
摘要:
内存瓶颈仍是训练大语言模型的主要限制因素。我们引入了Spectral Compact Training (SCT),通过永久截断SVD因子W = U diag(s) V^T替代密集权重矩阵,其中完整密集矩阵在训练或推理过程中从未被显式存储。梯度通过紧凑的谱因子通过标准反向传播流动,并在每个优化器步骤后通过QR分解将U、V回退到Stiefel流形。SCT在秩32时实现每MLP层高达199倍的内存减少,使70B参数架构能够在Steam Deck手持设备上完成完整训练步骤(7.2 GB峰值内存 vs. 1,245 GB for dense FP32 training with Adam)。在SmolLM2-1.7B上的秩扫描实验显示,所有测试秩都收敛到相同损失底部(~4.2-4.5),确定学习率调度而不是MLP秩是主要瓶颈。秩128在11.7倍MLP压缩下成为效率最佳点,具有最低的困惑度。GPU内存在秩32时减少46%,而训练吞吐量翻倍。
英文摘要:
The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。