arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.03260stat.MLcs.LG
收藏

通过局部二次逼近实现自动、动态和近乎最佳的学习率规范

Automatic, Dynamic, and Nearly Optimal Learning Rate Specification by Local Quadratic Approximation

Renmin University of China(中国人民大学);Peking University(北京大学)

Yingqiu Zhu, Yu Chen, Danyang Huang, Bo Zhang, Hansheng Wang

摘要:

在深度学习任务中,学习率决定了每次迭代中的更新步长,这在基于梯度的优化中起着关键作用。然而,在实践中确定适当的学习率通常是基于主观判断。在这项工作中,我们提出了一种新的优化方法的基础上局部二次近似(LQA)。在每个更新步骤中,给定梯度方向,我们通过学习率的标准二次函数局部近似损失函数。然后,我们提出了一个近似步骤,以获得一个计算效率高的方式接近最优的学习率。所提出的LQA方法具有三个重要特征。首先,在每个更新步骤中自动确定学习率。其次,根据当前损失函数值和参数估计值动态调整。第三,在梯度方向固定的情况下,所提出的方法在损失函数方面导致几乎最大的减少。已经进行了大量的实验,以证明所提出的LQA方法的优势。

英文摘要:

In deep learning tasks, the learning rate determines the update step size in each iteration, which plays a critical role in gradient-based optimization. However, the determination of the appropriate learning rate in practice typically replies on subjective judgement. In this work, we propose a novel optimization method based on local quadratic approximation (LQA). In each update step, given the gradient direction, we locally approximate the loss function by a standard quadratic function of the learning rate. Then, we propose an approximation step to obtain a nearly optimal learning rate in a computationally efficient way. The proposed LQA method has three important features. First, the learning rate is automatically determined in each update step. Second, it is dynamically adjusted according to the current loss function value and the parameter estimates. Third, with the gradient direction fixed, the proposed method leads to nearly the greatest reduction in terms of the loss function. Extensive experiments have been conducted to prove the strengths of the proposed LQA method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑