Practical Muon 在有限牛顿-舒尔茨迭代与内斯特罗夫动量下的收敛性
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
浏览论文内容
中文总结 AI 辅助
本文分析 Practical Muon 在有限牛顿-舒尔茨迭代与内斯特罗夫动量下的收敛性，在非凸随机优化中建立 $\mathcal O(T^{-1/4})$ 梯度范数界，无需对称噪声或奇异值下界，并验证五步迭代的标量界。
中文摘要 AI 辅助
Practical Muon 对每个参数矩阵分别维护动量，并执行少量固定次数的牛顿-舒尔茨迭代，通常还带有内斯特罗夫修正。我们在一个耦合的非凸目标上联合分析这些逐层有限步更新，而不是用精确极分解或单次全局正交化来替代它们。在梯度依赖的 $(\mathcal L_0,\mathcal L_1,q)$-光滑性以及具有有界逐层方差的条件下无偏随机梯度假设下，我们建立了期望平均 Frobenius 梯度范数的 $\mathcal O(T^{-1/4})$ 界。该分析保留了内斯特罗夫递推，且既不需要有界随机梯度、对称噪声，也不需要非零输出奇异值的一致正下界。当块数和问题常数固定时，其常数不包含显式的矩阵维数或秩因子。证明遵循一个下降不等式，并将动量跟踪误差分解为初始化、噪声和漂移三部分。对于原始的五步五次迭代，我们解析地验证了所需的标量映射界；该结果也允许满足相同界的步长相关系数。一个补充的核范数结果在更强的谱条件下量化了秩依赖性。消失速率使用耦合的学习率和动量调度，包括标准的单系数内斯特罗夫规则。
英文摘要
Practical Muon maintains momentum and performs a small, fixed number of Newton--Schulz iterations separately for each parameter matrix, often with a Nesterov correction. We analyze these layer-wise finite-step updates jointly on a coupled nonconvex objective, rather than replacing them by exact polar factors or one global orthogonalization. Under gradient-dependent $(\mathcal L_0,\mathcal L_1,q)$-smoothness and conditionally unbiased stochastic gradients with bounded layer-wise variance, we establish an $\mathcal O(T^{-1/4})$ bound on the expected average Frobenius gradient norm. The analysis retains the Nesterov recursion and requires neither bounded stochastic gradients, symmetric noise, nor a uniform positive lower bound on the nonzero output singular values. Its constants contain no explicit matrix-dimension or rank factors when the number of blocks and problem constants are fixed. The proof follows a descent inequality and a decomposition of the momentum tracking error into initialization, noise, and drift. For the original five-step quintic, we verify the required scalar-map bounds analytically; the result also allows step-dependent coefficients satisfying the same bounds. A complementary nuclear-norm result quantifies rank dependence under a stronger spectral condition. The vanishing rate uses coupled learning-rate and momentum schedules, including the standard single-coefficient Nesterov rule.
发表机构
- Pengcheng Laboratory（鹏城实验室）
机构由 AI 辅助整理，请以论文原文为准。
