arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.18528math.OCcs.LG
收藏

尺度不变神经网络优化:范数几何与重尾噪声

Tight Complexity of Muon under Heavy-Tailed Noise

Department of Industrial Engineering and Operations Research(工业工程与运营管理系)

Jiayu Zhang, Tianyi Lin

AI总结:

针对重尾噪声下的非凸随机优化,研究了尺度不变一阶方法的维度依赖下界,并提出了匹配上界的批处理Scion方法以及利用高阶光滑性的传输Scion方法。

摘要:

来自神经网络优化的一个日益增长的经验是,优化器的设计应尊重模型的参数化方式。尺度不变方法变得重要,因为其归一化的逐层更新不仅支持跨模型大小的超参数迁移,还能利用输入-输出矩阵范数几何。同时,深度学习中的随机梯度噪声通常远非亚高斯,可能表现出重尾。这些关键观察塑造了近期训练神经网络的算法原理,然而它们的联合理论后果仍未被充分探索。特别地,对于具有一般输入-输出矩阵范数的尺度不变方法,什么维度依赖是不可避免的,以及高阶光滑性是否能在重尾噪声下加速训练,尚不清楚。我们通过一般范数下 $\mathbb{R}^{m\times n}$ 上的非凸光滑随机优化来研究这些问题,目标是在 $p^{\mathrm{th}}$ 阶矩重尾噪声下达到 $\varepsilon$-稳定点。我们的第一个贡献是维度相关的下界:当 $\frac{\max\{m,n\}}{(\min\{m,n\})^2}$ 足够大时,任何具有谱范数的尺度不变一阶方法需要 $\Omega(\min\{m, n\}\varepsilon^{-\frac{3p-2}{p-1}})$ 次 oracle 调用。我们证明,具有谱范数的批处理 Scion 方法达到了匹配的上界 $O(\min\{m, n\}\varepsilon^{-\frac{3p-2}{p-1}})$。为了利用高阶光滑性,我们提出了一种传输 Scion 方法,并在范数为谱范数且 Hessian 矩阵 Lipschitz 连续时将界改进为 $O(\min\{m, n\}\varepsilon^{-\frac{5p-3}{2p-2}})$。最后,我们将实践启发式方法融入我们的传输方法,并在多种架构和模型大小上进行评估,展示了其在训练神经网络中的灵活性和兼容性。

英文摘要:

A growing lesson from neural network optimization is that optimizer design should respect how the model is parametrized. The layerwise input-output structure of neural networks motivates Muon and Scion, whose updates also support hyperparameter transfer. At the same time, stochastic gradient noise in deep learning is often far from sub-Gaussian and may exhibit heavy tails. These observations have shaped recent algorithmic principles for training neural networks, yet their joint theoretical consequences are underexplored. In particular, it is unclear what dimension dependence is unavoidable for gradient-based methods when the problem class is defined using an input-output norm and the stochastic gradient noise is heavy-tailed, and if higher-order smoothness can accelerate optimization. We study these questions through nonconvex smooth stochastic optimization over $\mathbb{R}^{m\times n}$ equipped with general norms and under $p^\mathrm{th}$-moment heavy-tailed noise, where the goal is to achieve an $ε$-stationary point in the dual norm. Our first contribution is a dimension-dependent lower bound: when $\max\{m,n\}$ is large enough, any method using only the stochastic gradients requires $Ω(\min\{m, n\}ε^{-\frac{3p-2}{p-1}})$ oracle calls for the problem class defined by the spectral norm, which is a common input-output matrix norm. We prove that a batched Muon method with the spectral norm can achieve the matching upper bound of $O(\min\{m, n\}ε^{-\frac{3p-2}{p-1}})$. To exploit higher-order smoothness, we propose a transported Muon method and improve the bound to $O(\min\{m, n\}ε^{-\frac{5p-3}{2p-2}})$ when the norm is spectral and the Hessian is Lipschitz. Finally, we incorporate practical heuristics into our transported method and evaluate it across multiple architectures and model sizes, showing its compatibility with neural network training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑