Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration. Since NS is only approximate, directions with small singular values fail to be orthonormalized. In Muon, NS is applied to the momentum matrix at every step, yet little is known about how the singular value spectrum of these momentum matrices behaves during training, or how that behavior changes with model size. We present the first systematic study of this question. Tracking singular value quantiles of the momentum buffer across layers in models ranging from 77M to 2.8B parameters, we observe a consistent picture: after a short burn-in, the quantiles stabilize at a value determined by the layer type and model size. These stabilization values follow remarkably clean power laws in model size, with layer-dependent exponents. Layers up to mid-late depth scale very mildly with model size $M$ (around $M^{-0.25}$), so the standard 5-step NS configuration used at academic scale will continue to orthonormalize them at much larger scales. Some of the late layers, however, scale much more aggressively (up to $M^{-0.96}$) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better-tuned coefficients. NS iterations are computationally expensive at scale; our laws give practitioners a principled, layer-aware recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter -- avoiding unnecessary computation without sacrificing update quality.
Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
Meituan Inc.(美团公司);The University of Hong Kong(香港大学);Pennsylvania State University(宾夕法尼亚州立大学);Peking University(北京大学);The Hong Kong University of Science and Technology(香港科学与技术大学)
Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院)
Reproducible scaling laws for contrastive language-image learning
LAION;UC Berkeley(加州大学伯克利分校);HuggingFace;University of Washington(华盛顿大学);Juelich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Juelich (FZJ)(于利希研究中心)
Institut für Theoretische Physik, Universität Heidelberg(海德堡大学理论物理研究所);Department of Physics, Harvard University(哈佛大学物理系);LPNHE, Sorbonne Université, Université Paris Cité, CNRS/IN2P3(巴黎大学LPNHE)
On the origin of neural scaling laws: from random graphs to natural language
Meta Superintelligence Labs, FAIR(Meta超智能实验室,FAIR);Department of Physics, University of Maryland, College Park(大学物理系,马里兰大学College Park分校);Joint Quantum Institute(联合量子研究所)
Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study
Department of Mathematics, Hong Kong Baptist University(香港 Baptist 大学数学系);Department of ACMS, University of Notre Dame(Notre Dame 大学ACMS系);School of Mathematics, Georgia Institute of Technology(佐治亚理工学院数学系);Department of Mathematics, UCLA(加州大学洛杉矶分校数学系)