arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.01356cs.LGstat.ML
收藏

Expectigrad:具有鲁棒收敛性质的快速随机优化

Expectigrad: Fast Stochastic Optimization with Robust Convergence Properties

Khoury College of Computer Sciences(科里计算机科学学院);Northeastern University(东北大学)

Brett Daley, Christopher Amato

摘要:

许多流行的自适应梯度方法,如Adam和RMSProp,都依赖于指数移动平均(EMA)来规范其步长。虽然EMA使这些方法对新的梯度信息高度敏感,但最近的研究表明,它也会导致至少一个凸优化问题的发散。我们提出了一种新的方法,称为Expectigrad,调整步长根据每个组件的所有历史梯度的未加权平均值,并计算一个偏差校正的动量项之间的分子和分母。我们证明,Expectigrad不能发散的每一个实例的优化问题,已知导致亚当发散。我们还建立了一个遗憾界一般的随机非凸设置,表明Expectigrad是不太容易受到梯度方差比现有的方法。在几个高维机器学习任务上测试Expectigrad,我们发现它通常比最先进的方法表现更好,几乎没有超参数调整。

英文摘要:

Many popular adaptive gradient methods such as Adam and RMSProp rely on an exponential moving average (EMA) to normalize their stepsizes. While the EMA makes these methods highly responsive to new gradient information, recent research has shown that it also causes divergence on at least one convex optimization problem. We propose a novel method called Expectigrad, which adjusts stepsizes according to a per-component unweighted mean of all historical gradients and computes a bias-corrected momentum term jointly between the numerator and denominator. We prove that Expectigrad cannot diverge on every instance of the optimization problem known to cause Adam to diverge. We also establish a regret bound in the general stochastic nonconvex setting that suggests Expectigrad is less susceptible to gradient variance than existing methods are. Testing Expectigrad on several high-dimensional machine learning tasks, we find it often performs favorably to state-of-the-art methods with little hyperparameter tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑