Muown 隐式执行角步长衰减
Muown Implicitly Performs Angular Step-size Decay
AI总结:
本文揭示 Muown 优化器的方向更新等价于归一化方向上的黎曼步长,其未归一化参数幅度仅调节角步长,据此提出显式角步长的 AngularMuown 方法,在 nanoGPT 竞赛和 Qwen2 等模型上表现更优。
摘要:
诸如 Muon 和 Muown 等矩阵感知优化器最近在预训练 Transformer 方面显示出强大的经验性能。特别是,Muown 将每个权重矩阵分解为行幅度和未归一化的方向变量,使用 Adam 更新前者,使用 Muon 更新后者。我们证明 Muown 的方向更新等价于归一化方向上的黎曼步长,而未归一化参数化的幅度仅调节角步长。这解释了 Muown 的步长稳定性,并建议使角步长显式化。由此产生的方法 AngularMuown 直接优化归一化方向,并使用与径向幅度更新解耦的可调度角乘数。AngularMuown 优于 Muown,并且在撰写本文时,一个初步版本在 modded nanoGPT 速度竞赛的每个优化器类别中领先。在 Qwen2-0.5B 和 1.1B 参数混合专家模型上的进一步实验证实该算法可扩展到小模型之外。该算法的实现可在此 https URL 获取。
英文摘要:
Matrix-aware optimizers such as Muon and Muown have recently shown strong empirical performance for pre-training Transformers. In particular, Muown separates each weight matrix into row magnitudes and an un-normalized direction variable, updating the former with Adam and the latter with Muon. We show that the directional update of Muown is equivalent to a Riemannian step on the normalized directions, while the magnitude of the un-normalized parameterization only modulates the angular step size. This explains the step-size stability of Muown and suggests making the angular step size explicit. The resulting method, AngularMuown, optimizes directly over the normalized directions and uses a schedulable angular multiplier decoupled from the radial magnitude update. AngularMuown improves over Muown and, at the time of writing, a preliminary version is leading the per-optimizer category of the modded nanoGPT speedrunning competition. Further experiments on Qwen2-0.5B, and 1.1B parameter mixture-of-experts models confirm the algorithm scales beyond small models. An implementation of the algorithm is available at https://github.com/fhueb/angular-muown
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。