arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.02831stat.MLcs.LGmath.OC
收藏

用于改善泛化能力的反相关噪声注入

Anticorrelated Noise Injection for Improved Generalization

ETH Zurich(苏黎世联邦理工学院);INRIA(法国国家信息与自动化研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);University of Oslo(奥斯陆大学);University of Basel(巴塞尔大学)

Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, Aurelien Lucchi

AI总结:

本文研究连续扰动梯度下降步骤的关联问题,提出Anti-PGD方法,通过注入反相关噪声使模型移动到更宽的极小值,实验和理论证明其泛化性能显著优于GD和标准PGD。

摘要:

将人工噪声注入梯度下降(GD)通常被用于提升机器学习模型的性能。通常,在此类扰动梯度下降(PGD)方法中会使用不相关噪声。然而,目前尚不清楚这是否最优,或者其他类型的噪声是否能提供更好的泛化性能。在本文中,我们深入研究关联连续 PGD 步骤扰动的问题。我们考虑了多种目标函数,并发现带有反相关扰动的 GD(“Anti-PGD”)的泛化性能显著优于 GD 和标准(不相关)PGD。为了支持这些实验发现,我们还推导了理论分析,证明 Anti-PGD 能够移动到更宽的极小值,而 GD 和 PGD 会停留在次优区域甚至发散。反相关噪声与泛化之间的这种新联系,为该领域开辟了利用噪声训练机器学习模型的新途径。

英文摘要:

Injecting artificial noise into gradient descent (GD) is commonly employed to improve the performance of machine learning models. Usually, uncorrelated noise is used in such perturbed gradient descent (PGD) methods. It is, however, not known if this is optimal or whether other types of noise could provide better generalization performance. In this paper, we zoom in on the problem of correlating the perturbations of consecutive PGD steps. We consider a variety of objective functions for which we find that GD with anticorrelated perturbations ("Anti-PGD") generalizes significantly better than GD and standard (uncorrelated) PGD. To support these experimental findings, we also derive a theoretical analysis that demonstrates that Anti-PGD moves to wider minima, while GD and PGD remain stuck in suboptimal regions or even diverge. This new connection between anticorrelated noise and generalization opens the field to novel ways to exploit noise for training machine learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑