arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2003.09503cs.LGstat.ML
收藏

基于事件的神经网络在线训练控制

Event-Based Control for Online Training of Neural Networks

Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立理工学院);GIPSA-lab(格勒诺布尔图像语音信号自动控制实验室)

Zilong Zhao, Sophie Cerf, Bogdan Robu, Nicolas Marchand

摘要:

卷积神经网络(CNN)已成为图像分类任务中最常用的方法。在训练过程中,学习率和梯度是影响模型收敛速度的两个关键参数。学习学习率策略是基于时间的,即随着时间单调衰减。最近的最先进的技术集中在自适应梯度算法,即亚当及其版本。在本文中,我们考虑一个在线学习的情况下,我们提出了两个基于事件的控制回路来调整学习率的经典算法E(指数)/PD(比例微分)控制。第一个基于事件的控制回路将被实现,以防止当模型接近最佳时学习率的突然下降。第二个基于事件的控制回路将根据学习速度决定何时切换到下一个数据批次。使用两个最先进的机器学习图像数据集(CIFAR-10和CIFAR-100)提供实验评估。实验结果表明,基于事件的E/PD算法优于原算法(最终精度更高,最终损失值更低),而基于双事件的E/PD算法可以加快训练过程,与现有算法相比,训练时间节省高达67%,性能更优。

英文摘要:

Convolutional Neural Network (CNN) has become the most used method for image classification tasks. During its training the learning rate and the gradient are two key factors to tune for influencing the convergence speed of the model. Usual learning rate strategies are time-based i.e. monotonous decay over time. Recent state-of-the-art techniques focus on adaptive gradient algorithms i.e. Adam and its versions. In this paper we consider an online learning scenario and we propose two Event-Based control loops to adjust the learning rate of a classical algorithm E (Exponential)/PD (Proportional Derivative)-Control. The first Event-Based control loop will be implemented to prevent sudden drop of the learning rate when the model is approaching the optimum. The second Event-Based control loop will decide, based on the learning speed, when to switch to the next data batch. Experimental evaluationis provided using two state-of-the-art machine learning image datasets (CIFAR-10 and CIFAR-100). Results show the Event-Based E/PD is better than the original algorithm (higher final accuracy, lower final loss value), and the Double-Event-BasedE/PD can accelerate the training process, save up to 67% training time compared to state-of-the-art algorithms and even result in better performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑