arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1612.01490stat.ML
收藏

Whiteout:深度神经网络中的高斯自适应噪声正则化

Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networks

Yinan Li, Fang Liu

摘要:

噪声注入是一种有效的抑制神经网络过拟合的方法。在dropout和shakeout中实施的Bernoulli NI程序与NN模型参数的$l_1$和$l_2$正则化有关。我们提出了白化,一个家庭NI正则化技术(NIRT)通过注入自适应高斯噪声在神经网络的训练。白化是第一个NIRT比施加广泛的$l_γ$稀疏正则化$(γ\in(0,2))$,而不必涉及$l_2$正则化。还可以扩展Whiteout以提供类似于自适应套索和组套索的正则化。我们建立了正则化效应的白化的广义线性模型的框架内封闭形式的惩罚条款,并表明白化稳定的训练神经网络的敏感性降低输入中的小扰动。我们建立了噪声扰动的经验损失函数(pelf)几乎必然收敛到理想损失函数(ilf),并且pelf的最小值与ilf的最小值是一致的。我们推导了pelf的尾界,以建立其最小化的实际可行性。在模拟和真实数据集中,whiteout在学习具有相对较小的训练集的NN和在大型训练集中的非劣效性方面优于Bernoulli NIRT,dropout和shakeout。这项工作是对深度神经网络中加性和乘性高斯NI的正则化效应的第一次深入的理论,方法和实践研究。

英文摘要:

Noise injection (NI) is an efficient technique to mitigate over-fitting in neural networks (NNs). The Bernoulli NI procedure as implemented in dropout and shakeout has connections with $l_1$ and $l_2$ regularization for the NN model parameters. We propose whiteout, a family NI regularization techniques (NIRT) through injecting adaptive Gaussian noises during the training of NNs. Whiteout is the first NIRT than imposes a broad range of the $l_γ$ sparsity regularization $(γ\in(0,2))$ without having to involving the $l_2$ regularization. Whiteout can also be extended to offer regularizations similar to the adaptive lasso and group lasso. We establish the regularization effect of whiteout in the framework of generalized linear models with closed-form penalty terms and show that whiteout stabilizes the training of NNs with decreased sensitivity to small perturbations in the input. We establish that the noise-perturbed empirical loss function (pelf) with whiteout converges almost surely to the ideal loss function (ilf), and the minimizer of the pelf is consistent for the minimizer of the ilf. We derive the tail bound on the pelf to establish the practical feasibility in its minimization. The superiority of whiteout over the Bernoulli NIRTs, dropout and shakeout, in learning NNs with relatively small-sized training sets and non-inferiority in large-sized training sets is demonstrated in both simulated and real-life data sets. This work represents the first in-depth theoretical, methodological, and practical examination of the regularization effects of both additive and multiplicative Gaussian NI in deep NNs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑