arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.01678cs.LGmath.OCstat.ML
收藏

记忆在随机优化中的作用

The Role of Memory in Stochastic Optimization

Department of Computer Science(计算机科学系);ETH Zürich(苏黎世联邦理工学院)

Antonio Orvieto, Jonas Kohler, Aurelien Lucchi

摘要:

如何保留过去梯度信息的选择极大地影响了最先进的随机优化方法的收敛特性,如Heavy-ball,Nesterov的动量,RMSprop和Adam。在此基础上,我们使用随机微分方程(SDES)明确研究记忆在基于梯度的算法中的作用。我们首先推导出一个通用的连续时间模型,该模型可以在确定性和随机设置中包含任意类型的记忆。我们提供了收敛保证,这是弱拟凸和二次增长的功能。然后,我们演示了如何离散化这个神经网络,以获得一个灵活的离散时间算法,可以实现从短期到长期的记忆板频谱。该算法不仅增加了从业者在算法选择上的自由度,而且在凸随机环境中具有比经典动量更好的稳定性。特别是,收敛不需要平均。有趣的是,我们的分析还提供了一个新的解释Nesterov的动量作为稳定的梯度放大,并强调了其不稳定的行为在(凸)随机设置的可能原因。此外,我们讨论了使用长期记忆的二阶矩估计的自适应方法,如亚当和RMSprop。最后,我们提供了一个广泛的实验研究不同类型的记忆在凸和非凸设置的效果。

英文摘要:

The choice of how to retain information about past gradients dramatically affects the convergence properties of state-of-the-art stochastic optimization methods, such as Heavy-ball, Nesterov's momentum, RMSprop and Adam. Building on this observation, we use stochastic differential equations (SDEs) to explicitly study the role of memory in gradient-based algorithms. We first derive a general continuous-time model that can incorporate arbitrary types of memory, for both deterministic and stochastic settings. We provide convergence guarantees for this SDE for weakly-quasi-convex and quadratically growing functions. We then demonstrate how to discretize this SDE to get a flexible discrete-time algorithm that can implement a board spectrum of memories ranging from short- to long-term. Not only does this algorithm increase the degrees of freedom in algorithmic choice for practitioners but it also comes with better stability properties than classical momentum in the convex stochastic setting. In particular, no iterate averaging is needed for convergence. Interestingly, our analysis also provides a novel interpretation of Nesterov's momentum as stable gradient amplification and highlights a possible reason for its unstable behavior in the (convex) stochastic setting. Furthermore, we discuss the use of long term memory for second-moment estimation in adaptive methods, such as Adam and RMSprop. Finally, we provide an extensive experimental study of the effect of different types of memory in both convex and nonconvex settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑