arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.03695cs.LGstat.ML
收藏

在球面上训练尺度不变的神经网络可以在三种情况下发生

Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

HSE University(高等经济大学);AIRI(AIRI研究所)

Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov, Dmitry Vetrov

摘要:

深度学习规范化技术(例如批量归一化)的基本属性是使预规范化参数具有尺度不变性。这些参数的固有域是单位球,因此它们的梯度优化动力学可以通过具有变化的有效学习率(ELR)的球形优化来表示,这是先前研究过的。然而,不同的ELR可能会掩盖内在损失景观结构的某些特征。在这项工作中,我们研究了使用固定的ELR直接在球体上训练尺度不变的神经网络的性质。我们发现三种制度,这种培训取决于ELR值:收敛,混沌平衡,发散。我们详细研究了这些机制,既对玩具示例进行了理论研究,又对真实尺度不变的深度学习模型进行了彻底的实证分析。每种机制都有独特的特征,反映了内在损失景观的特定属性,其中一些与以前对常规和尺度不变神经网络训练的研究有很强的相似之处。最后,我们展示了所发现的机制如何反映在规范化网络的传统训练中,以及如何利用它们来实现更好的最优解。

英文摘要:

A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying ELR may obscure certain characteristics of the intrinsic loss landscape structure. In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence. We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models. Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training. Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑