arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.10376cs.LGstat.ML
收藏

深度网络训练的两种方法

The Two Regimes of Deep Network Training

MIT(麻省理工学院)

Guillaume Leclerc, Aleksander Madry

摘要:

学习率调度对深度学习模型的性能有重要影响。尽管如此,时间表的选择往往是不确定的。我们的目标是发展一个精确的理解不同的学习率调度和适当的方式来选择它们的影响。为此,我们隔离了两个不同的训练阶段,第一个阶段,我们称之为“大步”制度,从优化的角度来看,表现出相当差的性能,但却是模型泛化的主要贡献者;后者,“小步”制度表现出更多的“凸状”优化行为,但孤立使用会产生泛化能力较差的模型。我们发现,通过分别对待这些机制,并使我们的训练算法专门针对其中的每一个,我们可以大大简化学习率调度。

英文摘要:

Learning rate schedule has a major impact on the performance of deep learning models. Still, the choice of a schedule is often heuristical. We aim to develop a precise understanding of the effects of different learning rate schedules and the appropriate way to select them. To this end, we isolate two distinct phases of training, the first, which we refer to as the "large-step" regime, exhibits a rather poor performance from an optimization point of view but is the primary contributor to model generalization; the latter, "small-step" regime exhibits much more "convex-like" optimization behavior but used in isolation produces models that generalize poorly. We find that by treating these regimes separately-and em specializing our training algorithm to each one of them, we can significantly simplify learning rate schedules.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑