arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.10597stat.MLcs.LG
收藏

统计自适应随机梯度法

Statistical Adaptive Stochastic Gradient Methods

Microsoft Research AI(微软研究院人工智能部门);Microsoft Research NExT(微软研究院新体验与技术部门);University of Texas at Austin(得克萨斯大学奥斯汀分校)

Pengchuan Zhang, Hunter Lang, Qiang Liu, Lin Xiao

摘要:

我们提出了一个统计自适应过程称为SALSA自动调度的学习率(步长)在随机梯度方法。SALSA首先使用平滑的随机线搜索过程来逐渐提高学习率,然后自动切换到统计方法来降低学习率。线搜索过程“预热”优化过程,减少了在设置初始学习率时昂贵的试验和错误的需要。降低学习率的方法是基于一种新的统计测试,当使用一个恒定的步长检测平稳性。与以前的工作不同,我们的测试适用于广泛的一类随机梯度算法,无需修改。该组合方法具有高度的鲁棒性和自主性,并且在我们对几个深度学习任务的实验中,它与最佳手动调整学习速率计划的性能相匹配。

英文摘要:

We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods. SALSA first uses a smoothed stochastic line-search procedure to gradually increase the learning rate, then automatically switches to a statistical method to decrease the learning rate. The line search procedure ``warms up'' the optimization process, reducing the need for expensive trial and error in setting an initial learning rate. The method for decreasing the learning rate is based on a new statistical test for detecting stationarity when using a constant step size. Unlike in prior work, our test applies to a broad class of stochastic gradient algorithms without modification. The combined method is highly robust and autonomous, and it matches the performance of the best hand-tuned learning rate schedules in our experiments on several deep learning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑