澄清洗发水:适应随机性和参数轨迹的谱下降
Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory
AI总结:
本文通过实验表明,Shampoo在权重矩阵上的应用比Muon更高效,其更新方式在期望上是时间平均的半正交,而非强制半正交性。
摘要:
利用神经网络中矩阵结构的优化器,如Shampoo和Muon,比元素级算法如Adam和Signum更高效。虽然在特定设置中,Shampoo和Muon减少到类似于Adam和Signum减少到符号下降的谱下降,但它们的一般关系和在受控设置下的相对数据效率仍不清楚。通过在语言模型上的广泛实验,我们证明Shampoo在token效率上优于Muon,类似于Adam在Signum上的优势。我们展示了Shampoo的更新应用于权重矩阵可以分解为适应的Muon更新。与此一致,Shampoo的益处可以仅归因于其应用于权重矩阵,挑战了不考虑参数形状的解释。这提供了一种新的视角,也避免了基于方差适应和白化的相关解释的缺陷:而不是强制半正交性如谱下降,Shampoo的更新在期望上是时间平均的半正交。
英文摘要:
Optimizers leveraging the matrix structure in neural networks, such as Shampoo and Muon, are more data-efficient than element-wise algorithms like Adam and Signum. While in specific settings, Shampoo and Muon reduce to spectral descent analogous to how Adam and Signum reduce to sign descent, their general relationship and relative data efficiency under controlled settings remain unclear. Through extensive experiments on language models, we demonstrate that Shampoo achieves higher token efficiency than Muon, mirroring Adam's advantage over Signum. We show that Shampoo's update applied to weight matrices can be decomposed into an adapted Muon update. Consistent with this, Shampoo's benefits can be exclusively attributed to its application to weight matrices, challenging interpretations agnostic to parameter shapes. This admits a new perspective that also avoids shortcomings of related interpretations based on variance adaptation and whitening: rather than enforcing semi-orthogonality as in spectral descent, Shampoo's updates are time-averaged semi-orthogonal in expectation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。