arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.14397stat.MLcs.LG
收藏

核域下ReLU网络的深等于浅

Deep Equals Shallow for ReLU Networks in Kernel Regimes

Center for Data Science, New York University(纽约大学数据科学中心);Inria - Département d’Informatique de l’École Normale Supérieure(法国国家信息与自动化研究所 - 巴黎高等师范学校计算机科学系);PSL Research University(巴黎文理研究大学)

Alberto Bietti, Francis Bach

摘要:

深度网络通常被认为在近似方面比浅层网络更具表达力。事实上,某些函数可以通过深度网络比浅层网络更有效地近似,然而,没有已知的用于学习这种深度模型的易处理算法。另外,最近的一项工作表明,使用梯度下降训练的深度网络在某种过度参数化的制度下可能表现得像(易处理的)核方法,其中核由架构和初始化决定,本文重点关注这种核的近似。我们表明,对于ReLU激活,从深层全连接网络中导出的内核与其浅层两层对应物具有基本相同的近似特性,即相应积分算子的相同特征值衰减。这突出了内核框架在理解这种深度架构的好处方面的局限性。我们的主要理论结果依赖于通过核函数的可微性来表征这种特征值衰减,这也很容易适用于球面上定义的其他内核的研究。

英文摘要:

Deep networks are often considered to be more expressive than shallow ones in terms of approximation. Indeed, certain functions can be approximated by deep networks provably more efficiently than by shallow ones, however, no tractable algorithms are known for learning such deep models. Separately, a recent line of work has shown that deep networks trained with gradient descent may behave like (tractable) kernel methods in a certain over-parameterized regime, where the kernel is determined by the architecture and initialization, and this paper focuses on approximation for such kernels. We show that for ReLU activations, the kernels derived from deep fully-connected networks have essentially the same approximation properties as their shallow two-layer counterpart, namely the same eigenvalue decay for the corresponding integral operator. This highlights the limitations of the kernel framework for understanding the benefits of such deep architectures. Our main theoretical result relies on characterizing such eigenvalue decays through differentiability properties of the kernel function, which also easily applies to the study of other kernels defined on the sphere.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑