利用线搜索定位随机非负关联梯度投影点自适应求解学习率
Resolving learning rates adaptively by locating Stochastic Non-Negative Associated Gradient Projection Points using line searches
摘要：
随机神经网络训练中的学习率目前是使用昂贵的手动或自动迭代调整来先验地确定训练的。本研究提出仅梯度线搜索来解决神经网络训练算法的学习率问题。训练过程中的随机子采样降低了计算成本，并允许优化算法在局部最小值上取得进展。然而，它也导致不连续的成本函数。最小化线搜索在这种情况下是无效的，因为它们使用消失导数（一阶最优性条件），其通常不存在于不连续的成本函数中，因此收敛到不连续性，而不是来自数据趋势的最小值。相反，我们使沿着搜索方向的候选解纯粹基于梯度信息，特别是通过从负到正的方向导数符号变化（非负关联梯度投影点（NN-GPP））。仅考虑从负到正的符号变化总是指示最小值，因此NN-GPP包含二阶信息。相反，消失梯度纯粹是一阶条件，其可以指示最小值、最大值或鞍点。这种洞察力允许算法的学习率被可靠地解析为沿着搜索方向的步长，从而提高收敛性能并消除否则昂贵的超参数。
英文摘要：
Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases computational cost and allows the optimization algorithms to progress over local minima. However, it also results in discontinuous cost functions. Minimization line searches are not effective in this context, as they use a vanishing derivative (first order optimality condition), which often do not exist in a discontinuous cost function and therefore converge to discontinuities as opposed to minima from the data trends. Instead, we base candidate solutions along a search direction purely on gradient information, in particular by a directional derivative sign change from negative to positive (a Non-negative Associative Gradient Projection Point (NN- GPP)). Only considering a sign change from negative to positive always indicates a minimum, thus NN-GPPs contain second order information. Conversely, a vanishing gradient is purely a first order condition, which may indicate a minimum, maximum or saddle point. This insight allows the learning rate of an algorithm to be reliably resolved as the step size along a search direction, increasing convergence performance and eliminating an otherwise expensive hyperparameter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
