arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.12191math.OCcs.LG
收藏

一种能量动量自适应梯度法

An Adaptive Gradient Method with Energy and Momentum

Iowa State University(爱荷华州立大学)

Hailiang Liu, Xuping Tian

摘要:

本文提出了一种基于梯度的随机目标函数优化算法。该方法可以被视为具有动量的SGD的变体,其配备有通过“能量”变量自动调整的自适应学习率。该方法实现简单,计算效率高,非常适合大规模机器学习问题。该方法具有无条件的能量稳定性的任何大小的基本学习率。在在线凸优化框架下,给出了收敛速度的遗憾界。在随机非凸条件下,我们还建立了算法到一个稳定点的能量依赖的收敛速度。此外,提供了一个充分条件来保证能量变量的正下限阈值。我们的实验表明,该算法收敛速度快,同时在训练深度神经网络时比SGD更好或更好地推广,并且与Adam相比也很有利。

英文摘要:

We introduce a novel algorithm for gradient-based optimization of stochastic objective functions. The method may be seen as a variant of SGD with momentum equipped with an adaptive learning rate automatically adjusted by an 'energy' variable. The method is simple to implement, computationally efficient, and well suited for large-scale machine learning problems. The method exhibits unconditional energy stability for any size of the base learning rate. We provide a regret bound on the convergence rate under the online convex optimization framework. We also establish the energy-dependent convergence rate of the algorithm to a stationary point in the stochastic non-convex setting. In addition, a sufficient condition is provided to guarantee a positive lower threshold for the energy variable. Our experiments demonstrate that the algorithm converges fast while generalizing better than or as well as SGD with momentum in training deep neural networks, and compares also favorably to Adam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑