Extending $μ$P: Spectral Conditions for Feature Learning Across Optimizers
Purdue University(普渡大学);Argonne National Laboratory(阿贡国家实验室)
Akshita Gupta, Marieme Ngom, Sam Foreman, Venkatram Vishwanath
备注:
10 main pages, 16 appendix pages and 17 figures; Amended version of the publication in 17th International OPT Workshop on Optimization for Machine Learning
Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization $(μ$P$)$ is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving $μ$P for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive $μ$P for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our $μ$P derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
University of Cambridge(剑桥大学);Flower Labs;Institute of Science and Technology Austria(奥地利科学技术研究所);University of Warwick(华威大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study
School of Computer and Cyber Sciences(计算机与电子科学学院);Augusta University(奥古斯塔大学);Department of Graduate Psychology(研究生心理学系);James Madison University(詹姆斯·麦迪逊大学);School of Computing(计算学院);University of Georgia(佐治亚大学)
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus(网络安全科学与技术学院,中山大学深圳校区);School of Computer Science, University of Science and Technology of China(计算机科学学院,中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学)
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏格兰先进研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Stanford University(斯坦福大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Arizona State University(亚利桑那州立大学);Nanyang Technological University(南洋理工大学);SB Intuitions Corp.(SB直觉公司)
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics(KU莱顿大学,电气工程系,动态系统、信号处理和数据分析中心);École Polytechnique Fédérale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS)(洛桑联邦理工学院(EPFL),信息与推断系统实验室);CVN, CentraleSupélec, Université Paris-Saclay, Inria(CVN,中央超导学院,巴黎-萨克雷大学,法国国家科学研究中心)
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学现代国家重点实验室(注:此处按原文直译,实际常见译名为“南京大学计算机软件新技术国家重点实验室”));University of Wollongong(卧龙岗大学)
Mathematics and Statistics Department, McGill University(麦吉尔大学数学与统计学系);Mathematics and Statistics Department, Université de Montréal(蒙特利尔大学数学与统计学系);Google DeepMind(谷歌DeepMind)
Muon with Spectral Guidance: Efficient Optimization for Scientific Machine Learning
Elmore Family School of Electrical and Computer Engineering(埃洛姆家族电气与计算机工程学院);Purdue University(普渡大学);Department of Mathematics(数学系);Department of Mathematics & School of Mechanical Engineering(数学系与机械工程学院)