arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12273cs.LGcs.NAmath.NA
收藏

采用自适应动量优化训练的神经网络参数空间中数字涨落的几何结构

Geometrical structures of digital fluctuations in parameter space of neural networks trained with adaptive momentum optimization

Institute for Information Transmission Problems, Russian Academy of Sciences(俄罗斯科学院信息传输问题研究所)

Igor V. Netay

AI总结:

本文研究了自适应动量优化在神经网络长期训练中的数值不稳定性,发现数值伪影会导致浅层窄网络发散,参数在空间中形成双扭曲螺旋,这一现象由一阶和二阶动量的数值扰动与弛豫振荡交替引起。

摘要:

我们展示了基于随机梯度的自适应动量优化神经网络的数值实验结果。这种广泛应用的优化方法已被证明具有收敛性和实际效率,但在长期训练中会出现数值不稳定。我们表明,数值伪影不仅在大规模模型中可观察到,最终也会导致浅层窄网络出现发散。我们通过对超过1600个训练了50000个epoch的神经网络进行实验来论证这一理论。局部观察表明,在稳定和不稳定的训练阶段,网络参数表现出相同的行为。参数的几何行为在参数空间中形成双扭曲螺旋,这是由数值扰动与随后的一阶和二阶动量值的弛豫振荡交替引起的。

英文摘要:

We present results of numerical experiments for neural networks with stochastic gradient-based optimization with adaptive momentum. This widely applied optimization has proved convergence and practical efficiency, but for long-run training becomes numerically unstable. We show that numerical artifacts are observable not only for large-scale models and finally lead to divergence also for case of shallow narrow networks. We argue this theory by experiments with more than 1600 neural networks trained for 50000 epochs. Local observations show presence of the same behavior of network parameters in both stable and unstable training segments. Geometrical behavior of parameters forms double twisted spirals in the parameter space and is caused by alternating of numerical perturbations with next relaxation oscillations in values for 1st and 2nd momentum.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑