arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.09541stat.MLcs.LGmath.PR
收藏

深度神经网络的谱复杂度

Spectral complexity of deep neural networks

RoMaDS - Department of Mathematics, University of Rome Tor Vergata(罗马第一大学数学系罗马塔 Vergata 分部)

Simmaco Di Lillo, Domenico Marinucci, Michele Salvi, Stefano Vigogna

AI总结:

该研究提出用极限高斯过程的角功率谱表征深度网络架构复杂度,定义相关随机变量序列并按深度渐近分布将网络分为三类,揭示不同激活函数的特征,且通过数值模拟验证了理论结果。

摘要:

众所周知,在所有层宽度趋于无穷的极限情况下,随机初始化的前馈全连接神经网络会弱收敛于各向同性高斯过程。本文提出利用该极限场的角功率谱来表征网络架构的复杂度。具体而言,我们定义了与角功率谱相关的随机变量序列,并根据深度趋于无穷时这些序列的渐近分布,给出了网络复杂度的完整表征。在此基础上,我们将神经网络分为低无序、稀疏和高无序三类;我们展示了该分类如何凸显标准激活函数的诸多不同特征,尤其是ReLU网络的稀疏性。我们的理论结果也通过数值模拟得到了验证。

英文摘要:

It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑