arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.05960cs.LGcs.CVstat.ML
收藏

基于局部弹性随机微分方程的深度学习动力学模拟

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

University of Pennsylvania(宾夕法尼亚大学)

Jiayao Zhang, Hua Wang, Weijie J. Su

摘要:

理解深度学习模型的训练动态可能是揭开这些模型有效性神秘面纱的必要一步。特别是,当使用随机梯度下降训练神经网络时,来自不同类别的数据如何在其特征空间中逐渐变得可分离?在这项研究中,我们使用一组随机微分方程(SDEs)来模拟深度学习训练期间特征的演变,每个随机微分方程对应于一个训练样本。作为我们建模策略中的一个重要组成部分,每个样本都包含一个漂移项,反映了输入端的反向传播对所有样本特征的影响。我们的主要发现揭示了关于类内影响的急剧相变现象:如果SDEs在某种意义上是局部弹性的,即对与输入相同的类的样本的影响更显著,则训练数据的特征变得线性可分离,这意味着训练损失消失;否则,无论训练时间多长,特征都不可分离。此外,在局部弹性的存在下,我们的SDES的分析表明,出现了一个简单的几何结构,称为神经崩溃的功能。总之,我们的研究结果揭示了局部弹性在神经网络的训练动力学中的决定性作用。我们证实了我们的理论分析与实验的合成数据集的几何形状和CIFAR-10。

英文摘要:

Understanding the training dynamics of deep learning models is perhaps a necessary step toward demystifying the effectiveness of these models. In particular, how do data from different classes gradually become separable in their feature spaces when training neural networks using stochastic gradient descent? In this study, we model the evolution of features during deep learning training using a set of stochastic differential equations (SDEs) that each corresponds to a training sample. As a crucial ingredient in our modeling strategy, each SDE contains a drift term that reflects the impact of backpropagation at an input on the features of all samples. Our main finding uncovers a sharp phase transition phenomenon regarding the {intra-class impact: if the SDEs are locally elastic in the sense that the impact is more significant on samples from the same class as the input, the features of the training data become linearly separable, meaning vanishing training loss; otherwise, the features are not separable, regardless of how long the training time is. Moreover, in the presence of local elasticity, an analysis of our SDEs shows that the emergence of a simple geometric structure called the neural collapse of the features. Taken together, our results shed light on the decisive role of local elasticity in the training dynamics of neural networks. We corroborate our theoretical analysis with experiments on a synthesized dataset of geometric shapes and CIFAR-10.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑