基于循环学习率的损失函数拓扑研究
Exploring loss function topology with cyclical learning rates
摘要:
我们提出了以前未报告的现象,而训练残差网络的观察和讨论。这项工作的目标是通过对这些新的经验结果的检验来更好地理解神经网络的本质。这些行为是通过应用周期性学习率(RNN)和线性网络插值来识别的。在这些行为中,违反直觉的是训练损失和快速训练的增加和减少。例如,我们展示了尽管使用了大的学习率,但与传统的训练相比,训练可以产生更高的测试准确率。复制这些结果的文件可在https://github.com/lnsmith54/exploring-loss上获得
英文摘要:
We present observations and discussion of previously unreported phenomena discovered while training residual networks. The goal of this work is to better understand the nature of neural networks through the examination of these new empirical results. These behaviors were identified through the application of Cyclical Learning Rates (CLR) and linear network interpolation. Among these behaviors are counterintuitive increases and decreases in training loss and instances of rapid training. For example, we demonstrate how CLR can produce greater testing accuracy than traditional training despite using large learning rates. Files to replicate these results are available at https://github.com/lnsmith54/exploring-loss
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。