arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.01124cs.LGcs.AI
收藏

Muon 作为残差连接

Muon as a Residual Connection

Hao Huang

AI总结:

提出 Muon 优化器可视为训练中的隐式残差连接,通过正交化更新牺牲即时梯度保真度以改善下游层表示保留,并在线性优化中验证其平衡局部下降与下游可用性的机制。

摘要:

Muon 最近已成为训练大型神经网络最有效的优化器之一,但其经验成功已从几个不同的角度得到解释。在本文中,我们提出一个简单的机制解释:Muon 可以理解为训练过程中的隐式残差连接。具体来说,正交化更新可以牺牲一些即时梯度保真度,同时改善下游层的表示保留。我们在受控的线性优化设置中研究这种权衡,其中 Muon 可以学习那些对局部目标拟合较慢但更容易被下游层利用的表示。我们的结果为 Muon 提供了一个概念性解释,并为平衡局部下降与下游可用性的优化器设计提供了视角。

英文摘要:

Muon has recently emerged as one of the most effective optimizers for training large neural networks, yet its empirical success has been explained from several different perspectives. In this paper, we propose a simple mechanistic interpretation: Muon can be understood as an implicit residual connection during training. Specifically, orthogonalizing the update can sacrifice some immediate gradient fidelity while improving representation preservation for downstream layers. We study this trade-off in controlled linear optimization settings, where Muon can learn representations that are slower to fit a local target but easier for downstream layers to exploit. Our results suggest a conceptual explanation for Muon and a design perspective for optimizers that balance local descent with downstream usability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑