arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1801.09136stat.ML
收藏

通过自适应在线学习率重新审视梯度下降

Gradient descent revisited via an adaptive online learning rate

Mathieu Ravaut, Satya Gorti

摘要:

任何梯度下降优化都需要选择学习率。随着模型越来越深,调整学习率很容易变得乏味,并且不一定会导致理想的收敛。我们提出了一种变化的梯度下降算法,其中的学习率是不固定的。相反,我们通过另一种梯度下降(一阶方法)或牛顿方法(二阶方法)来学习学习率本身。这样,可以优化任何机器学习算法的梯度下降。

英文摘要:

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, we learn the learning rate itself, either by another gradient descent (first-order method), or by Newton's method (second-order). This way, gradient descent for any machine learning algorithm can be optimized.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑