arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.24283cs.LGcs.AIcs.CL
收藏

驯服动量:通过低秩近似重新思考优化器状态

Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation

University of Science and Technology of China(中国科学技术大学);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)

Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan

AI总结:

LoRA-Pre通过低秩近似优化器状态,提升预训练和微调效率,实现内存节省与性能提升

摘要:

现代优化器如Adam和Muon在训练大语言模型中起着核心作用,但它们对一阶和二阶动量的依赖引入了显著的内存开销,这限制了可扩展性和计算效率。在本文中,我们将这些动量中使用的指数移动平均(EMA)重新解释为通过在线梯度流训练线性回归器。基于这一等价性,我们引入了LoRA-Pre,一种新的低秩优化器,用于高效的预训练。具体来说,LoRA-Pre通过将完整的动量矩阵分解为在线线性学习器中的紧凑低秩子空间,减少了优化器的内存开销,从而在保持优化性能的同时提高了内存效率。我们通过预训练Llama架构家族的模型(从60M到1B参数)来实证验证LoRA-Pre的有效性。LoRA-Pre在所有模型大小上均取得最高性能。值得注意的是,LoRA-Pre在秩效率方面表现出色,使用仅 baseline 方法的1/8 的秩即可达到可比或更优的结果。除了预训练,我们评估了LoRA-Pre在微调场景中的有效性。在相同秩的情况下,LoRA-Pre在微调任务中始终优于所有高效的微调基线。具体而言,与标准LoRA相比,LoRA-Pre在Llama-3.1-8B上实现了3.14个百分点的显著改进,在Llama-2-7B上实现了6.17个百分点的改进,验证了我们的方法在预训练和微调范式中的有效性。我们的代码可在https://github.com/mrflogs/LoRA-Pre上公开获取。

英文摘要:

Modern optimizers like Adam and Muon are central to training large language models, but their reliance on first- and second-order momenta introduces significant memory overhead, which constrains scalability and computational efficiency. In this work, we reframe the exponential moving average (EMA) used in these momenta as the training of a linear regressor via online gradient flow. Building on this equivalence, we introduce LoRA-Pre, a novel low-rank optimizer designed for efficient pre-training. Specifically, LoRA-Pre reduces the optimizer's memory footprint by decomposing the full momentum matrix into a compact low-rank subspace within the online linear learner, thereby maintaining optimization performance while improving memory efficiency. We empirically validate LoRA-Pre's efficacy by pre-training models from the Llama architecture family, scaling from 60M to 1B parameters. LoRA-Pre achieves the highest performance across all model sizes. Notably, LoRA-Pre demonstrates remarkable rank efficiency, achieving comparable or superior results using only 1/8 the rank of baseline methods. Beyond pre-training, we evaluate LoRA-Pre's effectiveness in fine-tuning scenarios. With the same rank, LoRA-Pre consistently outperforms all efficient fine-tuning baselines. Specifically, compared to standard LoRA, LoRA-Pre achieves substantial improvements of 3.14 points on Llama-3.1-8B and 6.17 points on Llama-2-7B, validating our approach's effectiveness across both pre-training and fine-tuning paradigms. Our code is publicly available at https://github.com/mrflogs/LoRA-Pre.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑