arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09405cs.LGcond-mat.dis-nnstat.ML
收藏

为什么要预热学习率?内在机制与改进方法

Why Warmup the Learning Rate? Underlying Mechanisms and Improvements

University of Maryland(马里兰大学)

Dayal Singh Kalra, Maissam Barkeshli

AI总结:

该研究揭示深度学习学习率预热的核心机制是让网络进入损失景观条件更优区域以承受更大目标学习率,还提出基于损失弹射的初始学习率选择方法与Adam方差初始化方案,可减少甚至消除预热需求。

摘要:

深度学习中通常会对学习率$\eta$进行预热,常采用在$\eta_{\text{init}}=0$与预设目标值$\eta_{\text{trgt}}$之间的线性调度策略。本文通过使用SGD和Adam的系统实验表明,预热的核心优势在于通过迫使网络进入损失景观中条件更优的区域,使其能够承受更大的$\eta_{\text{trgt}}$。这种处理更大$\eta_{\text{trgt}}$的能力让超参数调优更具鲁棒性,同时提升最终性能。我们揭示了预热阶段存在不同的运行机制,这取决于训练起始于渐进锐化阶段还是锐度降低阶段,而该阶段又由初始化和参数化方式决定。基于这些洞见,我们展示了如何利用损失弹射机制合理选择$\eta_{\text{init}}$,该机制可减少预热步数,在某些情况下可完全消除预热需求。我们还提出了一种Adam中方差的初始化方法,其能带来与预热类似的效果。

英文摘要:

It is common in deep learning to warm up the learning rate $η$, often by a linear schedule between $η_{\text{init}} = 0$ and a predetermined target $η_{\text{trgt}}$. In this paper, we show through systematic experiments using SGD and Adam that the overwhelming benefit of warmup arises from allowing the network to tolerate larger $η_{\text{trgt}}$ {by forcing the network to more well-conditioned areas of the loss landscape}. The ability to handle larger $η_{\text{trgt}}$ makes hyperparameter tuning more robust while improving the final performance. We uncover different regimes of operation during the warmup period, depending on whether training starts off in a progressive sharpening or sharpness reduction phase, which in turn depends on the initialization and parameterization. Using these insights, we show how $η_{\text{init}}$ can be properly chosen by utilizing the loss catapult mechanism, which saves on the number of warmup steps, in some cases completely eliminating the need for warmup. We also suggest an initialization for the variance in Adam which provides benefits similar to warmup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑