优化学习率调度与随机特征模型的缩放定律理论
Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model
AI总结：
本文研究了随机特征模型中最优学习率调度与缩放定律，通过分析易相和难相，提出优化调度方法并验证其在ResNet和GPT-2任务中的有效性。
摘要：
设置深度学习模型的学习率是成功训练的关键部分。本文探讨了使用随机梯度下降（SGD）训练的幂律随机特征模型的最优学习率调度模型。我们考虑了最优调度η_T^*(t)，其中t是当前迭代，T是训练时间跨度。该调度通过数值优化和最优控制理论分析得出。研究发现易相和难相两种模式，易相中调度呈多项式衰减，难相中调度类似预热-稳定-衰减。进一步探讨了学习率和批量大小的联合优化，并发现批量预热可提高易相训练时间。此外，还推导了动量参数β(t)的最优调度，并证明其在难相中提高了损失缩放指数。
英文摘要：
Setting the learning rate (LR) for a deep learning model is a critical part of successful training. Choosing LRs is often done empirically with trial and error. In this work, we explore a solvable model of optimal LR schedules for a powerlaw random feature model trained with stochastic gradient descent (SGD). We consider the optimal schedule $η_T^\star(t)$ where $t$ is the current iterate and $T$ is the training horizon. This schedule is computed both as a numerical optimization problem and also analytically using optimal control theory. Our analysis reveals two regimes which we term the easy phase and hard phase. In the easy phase the optimal schedule is a polynomial decay $η_T^\star(t) \simeq T^{-ξ} (1-t/T)^δ$ where $ξ$ and $δ$ depend on the properties of the features and task. In the hard phase, the optimal schedule resembles warmup-stable-decay with constant initial LR and annealing performed over a vanishing fraction of training steps. We investigate joint optimization of LR and batch size and find batch ramps can improve the wall-clock time in the easy phase. Beyond SGD, we derive optimal schedules for momentum parameter $β(t)$ and show that it improves the loss-scaling exponent in the hard phase. We compare our optimal schedule to various benchmarks including (1) optimal constant learning rates $η_T(t) \sim T^{-ξ}$ (2) optimal power laws $η_T(t) \sim T^{-ξ} t^{-χ}$, finding that our schedule achieves better rates than either of these. Our theory suggests that LR transfer across training horizon depends on the structure of the model and task. For ResNet image classification on CIFAR-5M, the learning curves exhibit hard-phase behavior where optimal base LRs are constant under sufficient annealing. GPT-2 style transformers trained in language modeling exhibit easy-phase behavior where optimal LRs shift even under annealing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
