arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.20937cs.LG
收藏

扩展μP:跨优化器的特征学习谱条件

Extending $μ$P: Spectral Conditions for Feature Learning Across Optimizers

Purdue University(普渡大学);Argonne National Laboratory(阿贡国家实验室)

Akshita Gupta, Marieme Ngom, Sam Foreman, Venkatram Vishwanath

AI总结:

本文提出了一种新的框架,用于推导更广泛优化器类别的μP,实现了跨优化器的零样本学习率转移和深度缩放参数化。

摘要:

为加速和扩展大型语言模型的训练,已经提出了多种自适应一阶和二阶优化方法的变种。这些优化过程的性能对超参数(HPs)的选择非常敏感,而为大规模模型调整这些超参数计算成本很高。最大更新参数化(μP)是一组缩放规则,旨在使最优的HPs独立于模型大小,从而允许在较小(计算成本更低)的模型上调整的HPs转移到训练更大的目标模型。尽管SGD和Adam在实践中表现出色,但为其他优化器推导μP具有挑战性,因为底层张量编程方法难以把握。基于最近引入的谱条件作为张量程序替代方案的工作,我们提出了一种新的框架,用于推导更广泛优化器类别的μP,包括AdamW、ADOPT、LAMB、Sophia、Shampoo和Muon。我们在多个基准模型上实现了我们的μP推导,并展示了上述优化器在增加模型宽度时的零样本学习率转移。此外,我们还提供了这些优化器深度缩放参数化的实证见解。

英文摘要:

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization $(μ$P$)$ is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving $μ$P for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive $μ$P for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our $μ$P derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑