arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1907.09008cs.CVcs.LGmath.OCstat.ML
收藏

signADAM:深度神经网络的学习置信度

signADAM: Learning Confidences for Deep Neural Networks

Xidian University(西安电子科技大学)

Dong Wang, Yicheng Liu, Wenwo Tang, Fanhua Shang, Hongying Liu, Qigong Sun, Licheng Jiao

摘要:

在本文中,我们提出了一种新的基于一阶梯度的算法来训练深度神经网络。我们首先介绍随机梯度的符号运算(如基于符号的方法,例如,”[10]“是的,是的。此外,为了使每个特征的拟合率更接近,我们定义了一个置信函数来区分梯度的不同分量,并将其应用到我们的算法中。它可以生成比现有算法更稀疏的梯度。我们称这个新算法为signADAM++。特别是,我们的两种算法都易于实现,可以加快各种深度神经网络的训练。signADAM++的动机是通过更新大而有用的梯度,而不管随机梯度中的无用信息,从最不同的样本中学习特征。我们还建立了理论上的收敛保证我们的算法。在各种数据集和模型上的实验结果表明,我们的算法产生了更好的性能比许多国家的最先进的算法,包括SIGN-SGD,SIGNUM和ADAM。我们还从多个角度分析了性能,包括损失景观,并开发了一种自适应方法,以进一步提高泛化能力。源代码可在https://github.com/DongWanginxdu/signADAM-Learn-by-Confidence上获得。

英文摘要:

In this paper, we propose a new first-order gradient-based algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in sign-based methods, e.g., SIGN-SGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we define a confidence function to distinguish different components of gradients and apply it to our algorithm. It can generate more sparse gradients than existing algorithms do. We call this new algorithm signADAM++. In particular, both our algorithms are easy to implement and can speed up training of various deep neural networks. The motivation of signADAM++ is preferably learning features from the most different samples by updating large and useful gradients regardless of useless information in stochastic gradients. We also establish theoretical convergence guarantees for our algorithms. Empirical results on various datasets and models show that our algorithms yield much better performance than many state-of-the-art algorithms including SIGN-SGD, SIGNUM and ADAM. We also analyze the performance from multiple perspectives including the loss landscape and develop an adaptive method to further improve generalization. The source code is available at https://github.com/DongWanginxdu/signADAM-Learn-by-Confidence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑