arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.05955cs.LGstat.ML
收藏

RSO:一种基于梯度自由采样的深度神经网络训练方法

RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

Rohun Tripathi, Bharat Singh

摘要:

我们提出了RSO(随机搜索优化),一种基于梯度自由马尔可夫链蒙特卡罗搜索的方法,用于训练深度神经网络。为此,RSO向深度神经网络中的权重添加扰动,并测试它是否减少了小批量的损失。如果这减少了损失,则更新权重,否则保留现有权重。令人惊讶的是,我们发现对每个权重重复这个过程几次就足以训练深度神经网络。与SGD的反向传播相比,RSO的权重更新数量要少一个数量级。RSO可以在每个步骤中进行积极的权重更新,因为没有学习率的概念。各个层的权重更新步骤也不与损失的幅度相耦合。RSO在MNIST和CIFAR-10数据集上的分类任务上进行了评估,深度神经网络为6到10层,准确率分别为99.1%和81.8%。我们还发现,更新后的权重只有5次,该算法获得了98%的分类准确率MNIST。

英文摘要:

We propose RSO (random search optimization), a gradient free Markov Chain Monte Carlo search based approach for training deep neural networks. To this end, RSO adds a perturbation to a weight in a deep neural network and tests if it reduces the loss on a mini-batch. If this reduces the loss, the weight is updated, otherwise the existing weight is retained. Surprisingly, we find that repeating this process a few times for each weight is sufficient to train a deep neural network. The number of weight updates for RSO is an order of magnitude lesser when compared to backpropagation with SGD. RSO can make aggressive weight updates in each step as there is no concept of learning rate. The weight update step for individual layers is also not coupled with the magnitude of the loss. RSO is evaluated on classification tasks on MNIST and CIFAR-10 datasets with deep neural networks of 6 to 10 layers where it achieves an accuracy of 99.1% and 81.8% respectively. We also find that after updating the weights just 5 times, the algorithm obtains a classification accuracy of 98% on MNIST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑