自动确定各种随机训练算法的学习率的仅限线性搜索
Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
摘要：
最近，在动态小批量子采样神经网络训练中，仅服从线性搜索和概率线性搜索重新引入了自适应确定学习速率的能力。然而，随机线搜索仍处于起步阶段，因此需要进行持续的调查。我们研究了仅服从线搜索不精确（GOLS-I）的应用，以自动确定选择流行的神经网络训练算法的学习率计划，包括NAG，Adagrad，Adadelta，Adam和LBFGS，具有许多浅，深和卷积神经网络架构，在不同的数据集上训练，具有各种损失函数。我们发现，GOLS-I的学习率时间表是有竞争力的手动调整的学习率，超过7个优化算法，三种类型的神经网络架构，23个数据集和两个损失函数。我们证明，算法，其中包括占主导地位的动量特性，是不是很适合与GOLS-I使用。然而，我们发现GOLS-I在自动确定超过15个数量级的学习率计划方面是有效的，对于大多数流行的神经网络训练算法，有效地消除了在神经网络训练中调整学习率计划的敏感超参数的需要。
英文摘要：
Gradient-only and probabilistic line searches have recently reintroduced the ability to adaptively determine learning rates in dynamic mini-batch sub-sampled neural network training. However, stochastic line searches are still in their infancy and thus call for an ongoing investigation. We study the application of the Gradient-Only Line Search that is Inexact (GOLS-I) to automatically determine the learning rate schedule for a selection of popular neural network training algorithms, including NAG, Adagrad, Adadelta, Adam and LBFGS, with numerous shallow, deep and convolutional neural network architectures trained on different datasets with various loss functions. We find that GOLS-I's learning rate schedules are competitive with manually tuned learning rates, over seven optimization algorithms, three types of neural network architecture, 23 datasets and two loss functions. We demonstrate that algorithms, which include dominant momentum characteristics, are not well suited to be used with GOLS-I. However, we find GOLS-I to be effective in automatically determining learning rate schedules over 15 orders of magnitude, for most popular neural network training algorithms, effectively removing the need to tune the sensitive hyperparameters of learning rate schedules in neural network training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
