Lotus: 通过随机低秩梯度投影与自适应子空间切换实现高效的LLM训练
Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
AI总结：
Lotus通过随机低秩梯度投影与自适应子空间切换，实现LLM训练的高效优化，减少30%训练时间和40%内存消耗。
摘要：
大规模模型的训练效率通常通过内存消耗、训练时间和模型性能来评估。当前方法往往在这些指标之间存在权衡，因为优化一个指标通常会损害至少另一个指标。解决这种权衡仍然是算法设计中的核心挑战。虽然GaLore通过在低秩子空间中更新梯度来实现内存高效的训练，但因其对梯度进行奇异值分解(SVD)过程而带来了相当的额外训练时间成本。在本文中，我们提出了Lotus方法，通过简单地修改投影过程来解决这种权衡。我们提出了一种准则，量化单位梯度的位移，以实现低秩梯度子空间之间的高效切换。实验结果表明，Lotus是最有效的方法，实现了30%的训练时间减少和40%的梯度和优化器状态内存消耗降低。此外，它在预训练和微调任务中均优于基线方法。
英文摘要：
Training efficiency in large-scale models is typically assessed through memory consumption, training time, and model performance. Current methods often exhibit trade-offs among these metrics, as optimizing one generally degrades at least one of the others. Addressing this trade-off remains a central challenge in algorithm design. While GaLore enables memory-efficient training by updating gradients in a low-rank subspace, it incurs a comparable extra training time cost due to the Singular Value Decomposition(SVD) process on gradients. In this paper, we propose Lotus, a method that resolves this trade-off by simply modifying the projection process. We propose a criterion that quantifies the displacement of the unit gradient to enable efficient transitions between low-rank gradient subspaces. Experimental results indicate that Lotus is the most efficient method, achieving a 30% reduction in training time and a 40% decrease in memory consumption for gradient and optimizer states. Additionally, it outperforms the baseline method in both pre-training and fine-tuning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
