arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1705.07795cs.LGmath.OCstat.ML
收藏

通过硬币投注训练无学习率的深度网络

Training Deep Networks without Learning Rates Through Coin Betting

Stony Brook University(石溪大学);Sapienza University of Rome(罗马大学)

Francesco Orabona, Tatiana Tommasi

摘要:

深度学习方法在许多应用场景中实现了最先进的性能。然而,这些方法需要大量的超参数调整,以实现最佳结果。特别是,调整学习率在随机优化过程中仍然是主要的瓶颈之一。在本文中,我们提出了一种新的深度网络随机梯度下降过程,不需要任何学习率设置。与以前的方法相反,我们不适应的学习率,也不使用假设的曲率的目标函数。相反,我们减少了优化过程中的一个游戏,赌一个硬币,并提出了一个学习率的最佳算法,这种情况下。理论上的收敛性证明了凸和拟凸函数和经验证据表明,我们的算法的优势,流行的随机梯度算法。

英文摘要:

Deep learning methods achieve state-of-the-art performance in many application scenarios. Yet, these methods require a significant amount of hyperparameters tuning in order to achieve the best results. In particular, tuning the learning rates in the stochastic optimization process is still one of the main bottlenecks. In this paper, we propose a new stochastic gradient descent procedure for deep networks that does not require any learning rate setting. Contrary to previous methods, we do not adapt the learning rates nor we make use of the assumed curvature of the objective function. Instead, we reduce the optimization process to a game of betting on a coin and propose a learning-rate-free optimal algorithm for this scenario. Theoretical convergence is proven for convex and quasi-convex functions and empirical evidence shows the advantage of our algorithm over popular stochastic gradient algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑