arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.25971cs.LG
收藏

通过解耦权重向量的幅度和方向改进神经网络训练

Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

Machine Learning and Optimization Lab, EPFL(EPFL机器学习与优化实验室)

Alexander Hägele, Alejandro Hernández-Cano, Atli Kosson, Martin Jaggi

AI总结:

提出幅度-方向(MD)解耦方法,将权重分解为固定范数的方向和可学习的幅度增益,以独立学习率更新,消除权重衰减和预热需求,在Adam和Muon优化器上均提升性能。

摘要:

现代神经网络训练依赖于如Adam和Muon等优化器,这些优化器将每个权重矩阵视为单个对象进行操作。然而,每个权重矩阵携带两个不同的量——\emph{幅度}和\emph{方向}——而所有在矩阵整体上步进的优化器都会耦合它们的动态:更新的方向变化取决于当前幅度,而幅度作为学习方向的副产品漂移,因此两者都不直接受学习率控制。因此,典型的训练依赖于诸如权重衰减和预热等辅助配方来保持大规模训练的稳定性,尽管这些方法仅间接调节耦合;其他近期方法则将权重约束到固定范数的球面上,但不添加可学习的幅度,将尺度控制仅留给归一化层。我们提出\emph{幅度-方向(MD)解耦},这是一种优化器修改,将每个权重分解为超球面上的固定范数方向和可学习的每行及每列幅度增益,以不同的学习率更新,同时模型仍然看到单个融合的权重张量。该方法与基础优化器无关,并消除了对权重衰减和预热的需求。在Adam和Muon上,MD解耦均改进了精心调优的基线,无需重新调优即可跨模型宽度转移最优学习率,并在大规模混合专家(MoE)模型上继续发挥作用。因此,将幅度和方向作为独立控制的量处理,可以产生更可预测的训练动态,并为现代优化器提供简单且广泛适用的改进。

英文摘要:

Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object. Yet every weight matrix carries two distinct quantities -- a \emph{magnitude} and a \emph{direction} -- and all optimizers stepping in the matrix as a whole couple their dynamics: the directional change from an update depends on the current magnitude, while the magnitude drifts as a byproduct of learning the direction. Then, neither is directly governed by the learning rate. Typical training therefore leans on surrounding recipes such as weight decay and warmup to keep learning stable at scale, though these regulate the coupling only indirectly. Other recent methods instead constrain the weight to a fixed-norm sphere, but add no learnable magnitude, leaving scale control to normalization layers alone. We propose \emph{Magnitude--Direction (MD) Decoupling}, an optimizer modification that factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates, all while the model still sees a single fused weight tensor. The method is agnostic to the base optimizer and removes the need for weight decay and warmup. Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts (MoE) models. Treating magnitude and direction as separately controlled quantities thus yields more predictable training dynamics and a simple, broadly applicable improvement to modern optimizers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑