arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.10498nlin.AOcs.LGmath.DSnlin.CDstat.ML
收藏

在线学习中的噪声诱导退化

Noise-induced degeneration in online learning

London Mathematical Laboratory(伦敦数学实验室);Osaka University(大阪大学)

Yuzuru Sato, Daiji Tsutsui, Akio Fujiwara

AI总结:

本研究针对神经网络梯度消失导致的平台现象,分析多层感知机退化子空间附近随机梯度下降的稳定性,发现噪声诱导同步的强平台现象及最优逃逸波动,揭示噪声诱导退化的普适性。

摘要:

为阐明由梯度消失导致的平台现象,本文分析了多层感知机中退化子空间附近随机梯度下降的稳定性。在Fukumizu-Amari模型(展现非平凡平台现象的最小多层感知机)的随机梯度下降过程中,我们证明:(1)多重退化子空间中存在吸引区域;(2)强平台现象作为一种噪声诱导同步现象出现,这在确定性梯度下降中未被观测到;(3)存在最优波动量,可使从退化子空间逃逸的时间最小化。本文观测到的噪声诱导退化现象,预计在各类基于神经网络的机器学习中普遍存在。

英文摘要:

In order to elucidate the plateau phenomena caused by vanishing gradient, we herein analyse stability of stochastic gradient descent near degenerated subspaces in a multi-layer perceptron. In stochastic gradient descent for Fukumizu-Amari model, which is the minimal multi-layer perceptron showing non-trivial plateau phenomena, we show that (1) attracting regions exist in multiply degenerated subspaces, (2) a strong plateau phenomenon emerges as a noise-induced synchronisation, which is not observed in deterministic gradient descent, (3) an optimal fluctuation exists to minimise the escape time from the degenerated subspace. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑