缪子的相：当缪子超越SignSGD时
Phases of Muon: When Muon Eclipses SignSGD
AI总结：
研究分析了缪子等谱优化器在高维矩阵最小二乘问题中的行为，揭示了SignSVD和SignSGD在不同批量大小下的收敛特性，发现其在数据协方差谱上有不同的预条件化行为，从而确定了三种相变情况。
摘要：
最近，缪子及相关谱优化器在可扩展的随机方法中表现出强大的经验性能，通常优于Adam。然而，其行为仍不明确。我们分析了包括缪子在内的随机谱优化器在高维矩阵最小二乘问题中的表现。我们推导出显式的确定性动力学，为研究学习行为提供了一个可处理的框架，重点在于缪子近似的随机SignSVD以及作为Adam代理的随机SignSGD。我们的分析表明，对于大批量大小，SignSVD对数据协方差谱进行平方根预条件化，而小批量大小下较小的特征模式行为类似于SGD，从而减缓收敛速度。我们对比了SignSGD，其在一般协方差下不进行预条件化且无相变，导致不同的最优学习率和收敛特性。两种方法在各向同性数据中匹配到常数因子，但在各向异性数据中表现不同。对具有数据指数α和目标指数β的幂律协方差模型的分析显示，在(α,β)平面中存在三种相：一种情况下SignSGD被普遍青睐，一种情况下SignSVD被普遍青睐，以及第三种情况下两种方法在性能上存在权衡。
英文摘要：
Muon and related spectral optimizers have recently demonstrated strong empirical performance as scalable stochastic methods, often outperforming Adam. Yet their behavior remains poorly understood. We analyze stochastic Muon, which orthogonalizes an exponential moving average of minibatch gradients, on a high-dimensional matrix-valued least squares problem, and we compare it with stochastic SignSGD, a simple baseline comparable to Adam. We derive explicit deterministic dynamics that provide a tractable framework for studying learning behavior. We show that Muon has non-linear sensitivity to the effective batch $B_{\rm eff}=B(1+β)/(1-β)$. With exact whitening and small $B_{\rm eff}$ Muon fails to resolve smaller eigenmodes, slowing down convergence. Momentum and finite Newton--Shulz iterations can act as a ridge which lessen this effect. This behavior is in contrast with SignSGD which has no transition, leading to different optimal learning rates, but with matching convergence rate up to a constant factor in the isotropic setting. For anisotropic data, large batch setting Muon performs a square-root preconditioning with respect to the data covariance spectrum while SignSGD provides no preconditioning for generic covariances. We analyze time to target loss in a power law covariance model and derive a phase diagram in the data exponent $α$ and target exponent $ζ$. Muon is favored for steep covariance spectra with rich targets ($α>2$, $ζ<α+1$), SignSGD is favored for flat spectra ($α<1$), and in other regions the winner depends on how the target loss scales with dimension. Our results shed light on why replacing Adam with Muon is not trivial even from a pure optimization perspective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
