Spectral gradient descent (SpecGD) orthogonalizes the matrix parameter updates and has inspired practical optimizers such as Muon. They often perform well in large language model (LLM) training, but their dynamics remain poorly understood. In the low-rank adaptation (LoRA) setting, where weight updates are parameterized as a product of two low-rank factors, we find a distinctive spectral phenomenon under Muon in LoRA fine-tuning of LLMs: singular values of the LoRA product show near-uniform growth across the spectrum, despite orthogonalization being performed on the two factors separately. Motivated by this observation, we analyze spectral gradient flow (SpecGF)-a continuous-time analogue of SpecGD-in a simplified LoRA-style matrix factorization setting and prove "equal-rate" dynamics: all singular values grow at equal rates up to small deviations. Consequently, smaller singular values attain their target values earlier than larger ones, sharply contrasting with the largest-first stepwise learning observed in standard gradient flow. Moreover, we prove that SpecGF in our setting converges to global minima from almost all initializations, provided the factor norms remain bounded; with $\ell_2$ regularization, we obtain global convergence. Lastly, we corroborate our theory with experiments in the same setting.
Preconditioning Benefits of Spectral Orthogonalization in Muon
Penn Department of Statistics and Data Science, Wharton School, University of Pennsylvania（宾夕法尼亚大学统计与数据科学系，沃顿商学院，宾夕法尼亚大学）；Yale Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系，耶鲁大学）；Department of Electrical and Systems Engineering, University of Pennsylvania（宾夕法尼亚大学电气与系统工程系，宾夕法尼亚大学）
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Imperial College London（帝国理工学院）；Technische Universität Berlin (TU Berlin)（柏林工业大学）；Ocean University of China（中国海洋大学）；The Hong Kong Polytechnic University（香港理工大学）
Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
Nanyang Technological University, Singapore（南洋理工大学，新加坡）；Indian Institute of Technology Bombay, India（印度理工学院班加罗尔，印度）；University of Sydney, Australia（悉尼大学，澳大利亚）；Microsoft India（微软印度）
