arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1909.03172cs.LGmath.OCstat.ML
收藏

理解噪声在训练神经网络中的重要性

Towards Understanding the Importance of Noise in Training Neural Networks

Peking University(北京大学);Georgia Tech(佐治亚理工学院)

Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, Tuo Zhao

摘要:

大量的经验证据已经证实,噪声在神经网络的有效训练中起着至关重要的作用。然而,背后的理论在很大程度上仍然是未知的。本文通过训练一个简单的两层卷积神经网络模型来研究这一基本问题。虽然训练这样一个网络需要解决一个非凸优化问题与虚假的局部最优和全局最优,我们证明了扰动梯度下降和扰动小批量随机梯度算法结合噪声退火是保证收敛到全局最优的多项式时间与任意初始化。这意味着噪声使算法能够有效地从虚假的局部最优值中逃脱。数值实验支持我们的理论。

英文摘要:

Numerous empirical evidence has corroborated that the noise plays a crucial rule in effective and efficient training of neural networks. The theory behind, however, is still largely unknown. This paper studies this fundamental problem through training a simple two-layer convolutional neural network model. Although training such a network requires solving a nonconvex optimization problem with a spurious local optimum and a global optimum, we prove that perturbed gradient descent and perturbed mini-batch stochastic gradient algorithms in conjunction with noise annealing is guaranteed to converge to a global optimum in polynomial time with arbitrary initialization. This implies that the noise enables the algorithm to efficiently escape from the spurious local optimum. Numerical experiments are provided to support our theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑