arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1908.06477cs.LGstat.ML
收藏

揭秘深度神经网络高精度训练的学习率策略

Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

Georgia Institute of Technology(佐治亚理工学院)

Yanzhao Wu, Ling Liu, Juhyun Bae, Ka-Ho Chow, Arun Iyengar, Calton Pu, Wenqi Wei, Lei Yu, Qi Zhang

摘要:

学习率(LR)是一个重要的超参数,用于调整深度神经网络(DNN)的有效训练。即使对于恒定学习率的基线,选择一个好的常数值来训练DNN也是非常重要的。动态学习率涉及在训练过程的各个阶段对LR值的多步调整,并提供高精度和快速收敛。然而,它们更难调谐。在本文中,我们提出了一个全面的研究13个学习率函数和相关的LR政策,通过检查他们的范围参数,步骤参数,和值更新参数。我们提出了一套指标来评估和选择LR政策,包括分类置信度,方差,成本和鲁棒性,并实现在LRBench,LR基准测试系统。LRBench可以帮助最终用户和DNN开发人员选择好的LR策略,避免糟糕的LR策略来训练他们的DNN。我们在开源深度学习框架Caffe上测试了LRBench,以展示LR策略的调优优化。通过广泛的实验评估,我们试图通过确定良好的LR政策,有效的LR值范围和步长LR更新时间表的LR政策,以揭开LR政策的调整。

英文摘要:

Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of the training process and offer high accuracy and fast convergence. However, they are much harder to tune. In this paper, we present a comprehensive study of 13 learning rate functions and their associated LR policies by examining their range parameters, step parameters, and value update parameters. We propose a set of metrics for evaluating and selecting LR policies, including the classification confidence, variance, cost, and robustness, and implement them in LRBench, an LR benchmarking system. LRBench can assist end-users and DNN developers to select good LR policies and avoid bad LR policies for training their DNNs. We tested LRBench on Caffe, an open source deep learning framework, to showcase the tuning optimization of LR policies. Evaluated through extensive experiments, we attempt to demystify the tuning of LR policies by identifying good LR policies with effective LR value ranges and step sizes for LR update schedules.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑