arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.04509cs.LGstat.ML
收藏

高维非凸优化问题的最优学习率调度

Optimal learning rate schedules in high-dimensional non-convex optimization problems

École Normale Supérieure(巴黎高等师范学院);Meta AI;EPFL(洛桑联邦理工学院)

Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli

摘要:

学习率调度被普遍用于加速和改进优化。许多不同的政策已被引入的经验基础上,并已开发的凸设置的理论分析。然而,在许多现实问题中,损失景观是高维和非凸的-结果是稀缺的情况下。在本文中,我们提出了第一个分析研究的学习率调度在这种设置中的作用,重点是朗之万优化的学习率衰减为$η(t)=t^{-β}$。我们首先考虑模型的损失是一个高斯随机函数的$N$维球($N\rightarrow \infty$),具有广泛的临界点。我们发现,要加快优化而不陷入鞍,必须选择一个衰减率$β<1$,相反的凸设置$β=1$通常是最优的。然后,我们在这个问题上加上一个要恢复的信号。在此设置中,动态分解为两个阶段:\emph{exploration}阶段,其中动态导航通过景观的粗糙部分,随后是\emph{convergence}阶段,其中检测到信号并且动态进入凸盆。在这种情况下,最佳的做法是在探索阶段保持较大的学习速率,以尽快逃离非凸区域,然后使用凸准则$β=1$快速收敛到解。最后,我们证明了我们的结论在涉及神经网络的常见回归任务中成立。

英文摘要:

Learning rate schedules are ubiquitously used to speed up and improve optimisation. Many different policies have been introduced on an empirical basis, and theoretical analyses have been developed for convex settings. However, in many realistic problems the loss-landscape is high-dimensional and non convex -- a case for which results are scarce. In this paper we present a first analytical study of the role of learning rate scheduling in this setting, focusing on Langevin optimization with a learning rate decaying as $η(t)=t^{-β}$. We begin by considering models where the loss is a Gaussian random function on the $N$-dimensional sphere ($N\rightarrow \infty$), featuring an extensive number of critical points. We find that to speed up optimization without getting stuck in saddles, one must choose a decay rate $β<1$, contrary to convex setups where $β=1$ is generally optimal. We then add to the problem a signal to be recovered. In this setting, the dynamics decompose into two phases: an \emph{exploration} phase where the dynamics navigates through rough parts of the landscape, followed by a \emph{convergence} phase where the signal is detected and the dynamics enter a convex basin. In this case, it is optimal to keep a large learning rate during the exploration phase to escape the non-convex region as quickly as possible, then use the convex criterion $β=1$ to converge rapidly to the solution. Finally, we demonstrate that our conclusions hold in a common regression task involving neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑