arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.04418cs.LGcs.AImath.OC
收藏

揭示大语言模型预训练中的流形约束

Demystifying Manifold Constraints in LLM Pre-training

Rice University(莱斯大学);Virginia Tech(弗吉尼亚理工大学);Columbia University(哥伦比亚大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Kang An, Jiaxiang Li, Donald Goldfarb, Shiqian Ma

AI总结:

本文通过激活尺度、旋转动力学和更新-权重比揭示流形约束与RMSNorm作用重叠,提出MACRO优化器,可消除权重衰减并缓解更新抵消,在低精度下稳健且具竞争力。

摘要:

矩阵优化器(如Muon)近期的成功表明,对动量进行特定归一化(如正交化和逐行归一化)有利于大语言模型训练的稳定性和加速。因此,近期多项研究建议对权重也进行归一化,从而引出了黎曼优化问题。尽管此类受约束的训练框架展现出优越性能,但显式约束权重的影响及其与现有稳定机制的相互作用仍未被充分理解。为弥补这一空白,我们通过激活尺度、旋转动力学和更新-权重比来研究流形约束训练动态。我们提出了一种名为MACRO的黎曼谱最陡下降优化器,并辅以半径选择原则作为我们的测试平台。我们的分析和数值实验揭示,RMSNorm和流形约束发挥重叠作用,并且当应用流形约束时,权重衰减可以被完全消除。通过控制更新-权重比,受约束训练显著缓解了更新抵消现象,实证表明MACRO对低精度计算具有鲁棒性,并且在标准大语言模型预训练中与现有算法具有竞争力。

英文摘要:

The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum, such as orthogonalization and row-wise normalization, benefits both the stability and acceleration of LLM training. Consequently, several recent studies have suggested that weights should also be normalized, leading to a Riemannian optimization problem. While such constrained training frameworks demonstrate superior performance, the effects of explicitly constraining weights, and their interaction with existing stabilization mechanisms, remain less understood. To bridge this gap, we study manifold constrained training dynamics through activation scales, rotational dynamics, and the update-to-weight ratio. We propose a Riemannian spectral steepest descent optimizer called MACRO, alongside a radius selection principle to serve as our testbed. Our analysis and numerical experiments reveal that RMSNorm and manifold constraints serve overlapping roles, and that weight decay can be completely eliminated when manifold constraints are applied. By controlling the update-to-weight ratio, constrained training significantly alleviates update cancellation, empirically demonstrating that MACRO is robust to low-precision computation and competitive with existing algorithms for standard LLM pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑