Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
University of Science and Technology of China(中国科学技术大学);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, Tieniu Tan
备注:
Camera-ready version. Accepted as Oral at ICLR 2026
Modern optimizers like Adam and Muon are central to training large language models, but their reliance on first- and second-order momenta introduces significant memory overhead, which constrains scalability and computational efficiency. In this work, we reframe the exponential moving average (EMA) used in these momenta as the training of a linear regressor via online gradient flow. Building on this equivalence, we introduce LoRA-Pre, a novel low-rank optimizer designed for efficient pre-training. Specifically, LoRA-Pre reduces the optimizer's memory footprint by decomposing the full momentum matrix into a compact low-rank subspace within the online linear learner, thereby maintaining optimization performance while improving memory efficiency. We empirically validate LoRA-Pre's efficacy by pre-training models from the Llama architecture family, scaling from 60M to 1B parameters. LoRA-Pre achieves the highest performance across all model sizes. Notably, LoRA-Pre demonstrates remarkable rank efficiency, achieving comparable or superior results using only 1/8 the rank of baseline methods. Beyond pre-training, we evaluate LoRA-Pre's effectiveness in fine-tuning scenarios. With the same rank, LoRA-Pre consistently outperforms all efficient fine-tuning baselines. Specifically, compared to standard LoRA, LoRA-Pre achieves substantial improvements of 3.14 points on Llama-3.1-8B and 6.17 points on Llama-2-7B, validating our approach's effectiveness across both pre-training and fine-tuning paradigms. Our code is publicly available at https://github.com/mrflogs/LoRA-Pre.
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏格兰先进研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Rethinking Prompt Optimizers: From Prompt Merits to Optimization
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore(南洋理工大学电子与电气工程学院);Home Team Science and Technology Agency, Singapore(新加坡科技局)
Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data
Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University(西安电子科技大学教育部协同智能系统重点实验室);Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所前沿人工智能研究中心);State Key Laboratory of ISN, School of Cyber Engineering, Xidian University(西安电子科技大学网络与信息安全学院综合业务网理论及关键技术国家重点实验室);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Department of Computing Technologies, Swinburne University of Technology(斯威本科技大学计算技术系)
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全国家重点实验室,计算技术研究所)
Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment
College of Computing and Data Science, Nanyang Technological University, Singapore.(南洋理工大学计算机与数据科学学院);Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada.(不列颠哥伦比亚大学电子与计算机工程系);Vector Institute, Canada(加拿大向量研究所);The Hong Kong Polytechnic University(香港理工大学)