arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.08093cs.LGcond-mat.dis-nncond-mat.stat-mechstat.ML
收藏

数据驱动的有效模型显示出液体般的深度学习

Data-driven effective model shows a liquid-like deep learning

Sun Yat-sen University(中山大学);PMI Lab(PMI实验室);School of Physics(物理学院)

Wenxuan Zou, Haiping Huang

摘要:

优化景观的几何结构被认为对支持深度神经网络学习的成功至关重要。直接计算两层以上的景观是困难的。因此,为了捕捉景观的全局视图,必须建立网络参数(或权重)空间的可解释模型。但是,到目前为止,该模型仍然缺乏。此外,二元突触的深度网络的前景仍然未知,这在鲁棒和节能的神经形态计算中起着关键作用。在这里,我们提出了一个统计力学框架,直接建立一个最少的结构化模型的高维权重空间,考虑现实的结构化数据,随机梯度下降训练,和计算深度的神经网络。我们还考虑了网络参数的数量是否超过了提供的训练数据的数量,即过参数化或欠参数化。我们的结构最少的模型表明,下参数化和过参数化的情况下的权重空间属于同一类,在这个意义上,这些权重空间是良好连接的,没有任何层次聚类结构。相比之下,浅网络具有破碎的权重空间,其特征在于不连续的相变,从而从高维几何的角度阐明了深度在深度学习中的益处。我们的有效模型还表明,在深度网络内部,存在一个类似液体的架构中心部分,从这个意义上说,这部分的权重尽可能随机地表现,提供算法含义。因此,我们的数据驱动模型提供了统计力学见解,了解为什么深度学习在高维权重空间方面非常有效,以及深度网络与浅层网络有何不同。

英文摘要:

The geometric structure of an optimization landscape is argued to be fundamentally important to support the success of deep neural network learning. A direct computation of the landscape beyond two layers is hard. Therefore, to capture the global view of the landscape, an interpretable model of the network-parameter (or weight) space must be established. However, the model is lacking so far. Furthermore, it remains unknown what the landscape looks like for deep networks of binary synapses, which plays a key role in robust and energy efficient neuromorphic computation. Here, we propose a statistical mechanics framework by directly building a least structured model of the high-dimensional weight space, considering realistic structured data, stochastic gradient descent training, and the computational depth of neural networks. We also consider whether the number of network parameters outnumbers the number of supplied training data, namely, over- or under-parametrization. Our least structured model reveals that the weight spaces of the under-parametrization and over-parameterization cases belong to the same class, in the sense that these weight spaces are well-connected without any hierarchical clustering structure. In contrast, the shallow-network has a broken weight space, characterized by a discontinuous phase transition, thereby clarifying the benefit of depth in deep learning from the angle of high dimensional geometry. Our effective model also reveals that inside a deep network, there exists a liquid-like central part of the architecture in the sense that the weights in this part behave as randomly as possible, providing algorithmic implications. Our data-driven model thus provides a statistical mechanics insight about why deep learning is unreasonably effective in terms of the high-dimensional weight space, and how deep networks are different from shallow ones.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑