Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers' matrix structure via orthogonalized momentum, showing superior performance in large language model training. We propose a new optimizer and a diagonal extension, NAMO and NAMO-D, providing the first principled integration of orthogonalized momentum with norm-based Adam-type noise adaptation. NAMO scales orthogonalized momentum using a single adaptive stepsize, preserving orthogonality while improving upon Muon at negligible additional cost. NAMO-D instead right-multiplies orthogonalized momentum by a diagonal matrix with clamped entries. This design enables neuron-wise noise adaptation and aligns with the common near block-diagonal Hessian structure. Under standard assumptions, we establish optimal convergence rates for both algorithms in the deterministic setting and show that, in the stochastic setting, their convergence guarantees adapt to the noise level of stochastic gradients. Experiments on pretraining GPT-2 models demonstrate improved performance of both NAMO and NAMO-D compared to the AdamW and Muon baselines, with NAMO-D achieving further gains over NAMO via an additional clamping hyperparameter that balances the competing goals of maintaining a well-conditioned update direction and leveraging fine-grained noise adaptation.
Muon Learns More Robust and Transferable Features than Adam
Yale University（耶鲁大学）；National University of Singapore（新加坡国立大学）；University of Chinese Academy of Sciences（中国科学院大学）；Academy of Mathematics and Systems Science, CAS（中国科学院数学与系统科学研究院）
本文通过鲁棒性和可迁移性视角，证明 Muon 优化器相比 Adam 和 SGD 能学习到更鲁棒、更可迁移的特征，并通过理论分析支持了经验发现。
Mathematics and Statistics Department, McGill University（麦吉尔大学数学与统计学系）；Mathematics and Statistics Department, Université de Montréal（蒙特利尔大学数学与统计学系）；Google DeepMind（谷歌DeepMind）
Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
Department of Computer Science University of British Columbia（计算机科学系不列颠哥伦比亚大学）；University of British Columbia Canada（不列颠哥伦比亚大学加拿大）；CIFAR AI Chair (Amii) &（CIFAR人工智能主席（Amii））；Department of Electrical and Computer Engineering University of British Columbia（电气与计算机工程系不列颠哥伦比亚大学）
Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab)（人工智能基础研究实验室（BRAIn Lab））；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（人工智能 Mohamed bin Zayed 大学）；Innopolis University（Innopolis 大学）
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
Department of Computer Science, Purdue University, West Lafayette, USA（计算机科学系，普渡大学，西拉法叶，美国）；Edwardson School of Industrial Engineering, Purdue University, West Lafayette, USA（工业工程学院，普渡大学，西拉法叶，美国）
An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
Department of Computer Science, George Mason University（乔治·马歇尔大学计算机科学系）；Center for Cosmology and Particle Physics, New York University（纽约大学宇宙与粒子物理中心）；Center for Computational Mathematics, Flatiron Institute（Flatiron研究所计算数学中心）
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University（人工智能微电子实验室、人工智能学院、计算机科学学院、上海交通大学）；Xiaohongshu Inc.（小红书公司）
TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers
Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country；Department of Combinatorics；Optimization, University of Waterloo, Waterloo, Canada；Huawei Canadian Research Institute, Canada；McGill University \& Mila-Quebec AI Institue, Canada
Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；School of Data Science, Fudan University（复旦大学数据科学学院）
Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
Department of Electrical and Computer Engineering, University of Minnesota, USA（电气与计算机工程系，明尼苏达大学，美国）；School of Mathematics and Statistics, University of Sydney, Australia（数学与统计学学院，悉尼大学，澳大利亚）
