arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04058cs.LGcs.AI
收藏

Muon的谱缩放定律

Spectral Scaling Laws of Muon

MIT(麻省理工学院)

Gagik Magakyan, Pablo Parrilo, Asuman Ozdaglar

AI总结:

本文系统研究了Muon优化器中动量矩阵奇异值谱随模型大小的缩放行为,发现其遵循幂律,并据此提出层感知的牛顿-舒尔茨迭代配置选择方法以减少计算开销。

摘要:

正交归一化更新规则已迅速成为训练大型语言模型的主流优化器选择,最近的开源最先进模型采用了Muon。为了保持这些更新的可处理性,Muon使用牛顿-舒尔茨(NS)迭代执行正交归一化。由于NS只是近似,小奇异值的方向无法被正交归一化。在Muon中,NS每一步都应用于动量矩阵,然而关于这些动量矩阵的奇异值谱在训练过程中如何行为,以及该行为如何随模型大小变化,我们知之甚少。我们首次系统研究了这一问题。通过追踪从77M到2.8B参数模型中各层动量缓冲区的奇异值分位数,我们观察到一致的现象:在短暂的预热后,分位数稳定在一个由层类型和模型大小决定的值上。这些稳定值随模型大小呈现出非常清晰的幂律,且指数依赖于层。中后深度的层随模型大小$M$的缩放非常温和(约$M^{-0.25}$),因此学术规模下使用的标准5步NS配置将在更大规模下继续对它们进行正交归一化。然而,某些后期层的缩放更为激进(高达$M^{-0.96}$),在前沿规模下将落入NS失效区域,除非使用更多NS迭代或更好调整的系数。NS迭代在规模上计算成本高昂;我们的定律为从业者提供了一种有原则的、层感知的配方,用于选择最小的NS配置,该配置仍能正交归一化重要的方向——在不牺牲更新质量的情况下避免不必要的计算。

英文摘要:

Orthonormalized update rules have rapidly become a leading choice of optimizer for training large language models, with recent open-source state-of-the-art models adopting Muon. To keep these updates tractable, Muon performs the orthonormalization with the Newton--Schulz (NS) iteration. Since NS is only approximate, directions with small singular values fail to be orthonormalized. In Muon, NS is applied to the momentum matrix at every step, yet little is known about how the singular value spectrum of these momentum matrices behaves during training, or how that behavior changes with model size. We present the first systematic study of this question. Tracking singular value quantiles of the momentum buffer across layers in models ranging from 77M to 2.8B parameters, we observe a consistent picture: after a short burn-in, the quantiles stabilize at a value determined by the layer type and model size. These stabilization values follow remarkably clean power laws in model size, with layer-dependent exponents. Layers up to mid-late depth scale very mildly with model size $M$ (around $M^{-0.25}$), so the standard 5-step NS configuration used at academic scale will continue to orthonormalize them at much larger scales. Some of the late layers, however, scale much more aggressively (up to $M^{-0.96}$) and will fall into the NS failure regime at frontier scale unless one uses more NS iterations or better-tuned coefficients. NS iterations are computationally expensive at scale; our laws give practitioners a principled, layer-aware recipe for choosing the minimum NS configuration that still orthonormalizes the directions that matter -- avoiding unnecessary computation without sacrificing update quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑