arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.04005cs.LG
收藏

迭代正交化缩放定律

Iterative Orthogonalization Scaling Laws

Devan Selvaraj

AI总结:

本文研究muon优化器中迭代正交化过程在大规模下的奇异值收缩问题,通过理论和随机矩阵实验揭示其缩放行为,但未给出解决方案。

摘要:

muon优化器近来作为可能替代似乎无处不在的Adam优化器而备受关注。最近,已有工作致力于记录muon下超参数(如权重衰减和学习率)的缩放定律。然而,在更大规模下,muon中的迭代正交化过程可能面临一个问题:随机矩阵的奇异值会随规模增大而缩小。本文从理论和实验两方面在随机矩阵上展示了这一缩放行为,但并未提出应对措施。

英文摘要:

The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑