arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.07613cs.LG

随机性助力严谨性：一座桥接理论与深度学习实践的概率学习率调度器

Randomness Helps Rigor: A Probabilistic Learning Rate Scheduler Bridging Theory and Deep Learning Practice

IIT Madras（印度理工学院马德拉斯分校）

Dahlia Devapriya, Thulasi Tholeti, Janani Suresh, Sheetal Kalyani

备注：
We have changed the title to better reflect our work and some parts of our manuscript for better readability
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有学习率调度器缺乏收敛性理论证明的问题，提出一种具备可证明收敛保证的概率学习率调度器PLRS，在多数据集与网络架构上验证其性能优于或持平现有最优调度器。

摘要：

学习率调度器在实践中已被证实可有效加速学习算法的收敛，但其收敛至最小值的特性尚未得到理论证明。这一难题主要源于：传统收敛分析要求学习率单调递减（或保持恒定），而调度器采用的学习率在训练轮次中往往呈现升降交替的变化。本研究旨在弥合这一理论与实践的鸿沟，提出一种不满足单调递减条件、且具备可证明收敛保证的概率学习率调度器（PLRS）。为验证本研究在现代应用中的相关性与实用性，我们在ResNet、WRN、VGG、DenseNet等深度神经网络架构上，结合CIFAR-10、CIFAR-100及Tiny ImageNet数据集开展了实验。结果表明，PLRS在收敛速度与准确率方面的表现与现有最先进的学习率调度器相当，甚至更优。例如，在CIFAR-100数据集上训练ResNet-110时，我们的方法在分类准确率上比当前最优的knee调度器高出1.56%。此外，在Tiny ImageNet数据集上使用ResNet-50架构时，我们的方法收敛稳定性显著优于余弦调度器，且分类准确率优于现有各类调度器。

英文摘要：

Learning rate schedulers have shown great success in speeding up the convergence of learning algorithms in practice. However, their convergence to a minimum has not been proven theoretically. This difficulty mainly arises from the fact that, while traditional convergence analysis prescribes to monotonically decreasing (or constant) learning rates, schedulers opt for rates that often increase and decrease through the training epochs. In this work, we aim to bridge the gap by proposing a probabilistic learning rate scheduler (PLRS) that does not conform to the monotonically decreasing condition, with provable convergence guarantees. To cement the relevance and utility of our work in modern day applications, we show experimental results on deep neural network architectures such as ResNet, WRN, VGG, and DenseNet on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. We show that PLRS performs as well as or better than existing state-of-the-art learning rate schedulers in terms of convergence as well as accuracy. For example, while training ResNet-110 on the CIFAR-100 dataset, we outperform the state-of-the-art knee scheduler by $1.56\%$ in terms of classification accuracy. Furthermore, on the Tiny ImageNet dataset using ResNet-50 architecture, we show a significantly more stable convergence than the cosine scheduler and a better classification accuracy than the existing schedulers.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑