随机优化问题的概率激励学习率自适应算法
A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization
摘要：
机器学习从业者在寻找优化算法的合适学习率方面投入了大量的人工和计算资源。我们提供了一个概率的动机，在高斯推断，流行的随机一阶方法。作为一个重要的特例，它恢复了具有一般度量的Polyak步骤。推理允许我们将学习率与无量纲量相关联，该无量纲量可以在训练期间通过控制算法自动适应。所得到的元算法被证明在应用于深度学习基准问题时，在大范围的初始值中以稳健的方式适应学习率。
英文摘要：
Machine learning practitioners invest significant manual and computational resources in finding suitable learning rates for optimization algorithms. We provide a probabilistic motivation, in terms of Gaussian inference, for popular stochastic first-order methods. As an important special case, it recovers the Polyak step with a general metric. The inference allows us to relate the learning rate to a dimensionless quantity that can be automatically adapted during training by a control algorithm. The resulting meta-algorithm is shown to adapt learning rates in a robust manner across a large range of initial values when applied to deep learning benchmark problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
