arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.15623stat.MLcs.LGmath.FA
收藏

多层ReLU网络的Banach空间:函数表示、逼近理论和梯度下降动力学

On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics

Princeton University(普林斯顿大学)

Weinan E, Stephan Wojtowytsch

摘要:

我们为有限深度$L$和无限宽度的ReLU神经网络开发了Banach空间。该空间包含所有有限的全连通$L$层网络和它们的$L^2$ -限制对象在自然路范数的界下。在此规范下,$L$层网络空间中的单位球具有较低的拉德马赫复杂度,从而具有良好的泛化性能。在这些空间中的函数可以近似的多层神经网络与维数无关的收敛速度。这项工作的关键是一种新的方式来表示功能的某种形式的期望,由多层神经网络的动机。这种表示允许我们为机器学习定义一类新的连续模型。我们表明,这种方式定义的梯度流是自然的连续模拟的梯度下降动态相关联的多层神经网络。我们表明,在这种连续的梯度流动力学的路径范数增加最多项式。

英文摘要:

We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑