arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1905.09899cs.LGmath.OCstat.ML
收藏

自适应:深度学习中更快的训练和更好的泛化

Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning

Hong Kong University of Science and Technology(香港科技大学)

Shuai Zheng, James T. Kwok

摘要:

具有坐标方向自适应步长的随机方法(如RMSprop和Adam)已广泛用于训练深度神经网络。尽管它们的收敛速度很快,但它们的泛化能力比随机梯度下降差。在本文中,通过重新审视Adagrad的设计,我们建议将网络参数分成块,并使用分块自适应步长。直观地说,分块自适应比对单个坐标的自适应更不积极,并且可以在自适应和泛化之间取得更好的平衡。我们从理论上表明,提出的块自适应梯度下降具有可比的收敛速度,其对应的坐标明智的自适应步长,但更快的一些常数。我们还研究了它的一致稳定性,并表明,分块自适应可以导致较低的泛化误差比坐标式自适应。实验结果表明,分块自适应梯度下降收敛速度快,提高了Nesterov的加速梯度和亚当的泛化性能。

英文摘要:

Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network parameters into blocks, and use a blockwise adaptive stepsize. Intuitively, blockwise adaptivity is less aggressive than adaptivity to individual coordinates, and can have a better balance between adaptivity and generalization. We show theoretically that the proposed blockwise adaptive gradient descent has comparable convergence rate as its counterpart with coordinate-wise adaptive stepsize, but is faster up to some constant. We also study its uniform stability and show that blockwise adaptivity can lead to lower generalization error than coordinate-wise adaptivity. Experimental results show that blockwise adaptive gradient descent converges faster and improves generalization performance over Nesterov's accelerated gradient and Adam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑