BFE和AdaBFE:随机优化学习率自动化的新方法
BFE and AdaBFE: A New Approach in Learning Rate Automation for Stochastic Optimization
摘要:
本文提出了一种基于梯度的自动调整学习率的优化方法。该方法可用于设计非自适应学习率和自适应学习率。首先介绍了非自适应学习率优化方法:二进制前向探索(Binary Forward Exploration,BFE),然后提出了相应的自适应每参数学习率优化方法:自适应BFE(Adaptive BFE,AdaBFE)。该方法可以作为一种基于随机梯度下降(SGD)算法优化学习率的替代方法,除了当前的非自适应学习率方法如SGD、动量、Nesterov和自适应学习率方法如AdaGrad、AdaDelta、Adam.开发这种方法的目的不是为了击败其他方法的基准,而只是为了提供一个不同的视角来优化梯度下降方法,尽管在下面的章节中将与以前的方法进行一些比较研究。这种方法有望成为启发式的,或启发研究人员改进基于梯度的优化与以前的方法相结合。
英文摘要:
In this paper, a new gradient-based optimization approach by automatically adjusting the learning rate is proposed. This approach can be applied to design non-adaptive learning rate and adaptive learning rate. Firstly, I will introduce the non-adaptive learning rate optimization method: Binary Forward Exploration (BFE), and then the corresponding adaptive per-parameter learning rate method: Adaptive BFE (AdaBFE) is possible to be developed. This approach could be an alternative method to optimize the learning rate based on the stochastic gradient descent (SGD) algorithm besides the current non-adaptive learning rate methods e.g. SGD, momentum, Nesterov and the adaptive learning rate methods e.g. AdaGrad, AdaDelta, Adam... The purpose to develop this approach is not to beat the benchmark of other methods but just to provide a different perspective to optimize the gradient descent method, although some comparative study with previous methods will be made in the following sections. This approach is expected to be heuristic or inspire researchers to improve gradient-based optimization combined with previous methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。