AMSGrad在非凸优化中的一个乐观加速算法
An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
摘要：
我们提出了AMSGrad的一个新变体，AMSGrad是一种流行的基于自适应梯度的优化算法，广泛用于训练深度神经网络。我们的算法添加了有关连续小批量梯度序列的先验知识，并利用其底层结构使梯度序列可预测。通过利用乐观在线学习的可预测性和思想，该算法可以加快收敛速度，提高样本效率。在建立了一个更严格的上限在一些凸性条件下的遗憾，我们提供了一个免费的看法，我们的算法，概括了离线和随机版本的非凸优化。在非凸的情况下，我们建立了一个独立于初始化的非渐近收敛界。我们通过数值实验说明了几种深度学习模型的实际加速。
英文摘要：
We propose a new variant of AMSGrad, a popular adaptive gradient based optimization algorithm widely used for training deep neural networks. Our algorithm adds prior knowledge about the sequence of consecutive mini-batch gradients and leverages its underlying structure making the gradients sequentially predictable. By exploiting the predictability and ideas from optimistic online learning, the proposed algorithm can accelerate the convergence and increase sample efficiency. After establishing a tighter upper bound under some convexity conditions on the regret, we offer a complimentary view of our algorithm which generalizes the offline and stochastic version of nonconvex optimization. In the nonconvex case, we establish a non-asymptotic convergence bound independently of the initialization. We illustrate the practical speedup on several deep learning models via numerical experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
