基于位移凸性的双层神经网络分析
Analysis of a Two-Layer Neural Network via Displacement Convexity
摘要：
通过使用大量$N$的“简单”成分的线性组合来拟合函数是统计学习中最富有成效的想法之一。这个想法是各种方法的核心，从双层神经网络到内核回归，再到boosting。一般来说，由此产生的风险最小化问题是非凸的，并通过梯度下降或其变体来解决。不幸的是，很少有人知道这些方法的全局收敛性。在这里，我们考虑的问题，学习凹函数$f$上的一个紧凑的凸域$Ω\subseteq {\mathbb R}^d$，使用线性组合的“凹凸”组件（神经元）。要拟合的参数是$N$凸块的中心，并且由此产生的经验风险最小化问题是高度非凸的。我们证明，在极限的神经元的数量发散，梯度下降的演化收敛到Wasserstein梯度流的概率分布的空间中的$Ω$。此外，当凸块宽度$δ$趋向于$0$时，该梯度流具有作为粘性多孔介质方程的极限。值得注意的是，通过这种梯度流优化的成本函数表现出一种特殊的属性，称为位移凸性，这意味着$N\to\infty$，$δ\to 0$的指数收敛速度。令人惊讶的是，这种渐近理论似乎很好地捕捉了$δ, N$的中等值的行为。解释这一现象，并了解依赖于$δ,N$的定量方式仍然是一个突出的挑战。
英文摘要：
Fitting a function by using linear combinations of a large number $N$ of `simple' components is one of the most fruitful ideas in statistical learning. This idea lies at the core of a variety of methods, from two-layer neural networks to kernel regression, to boosting. In general, the resulting risk minimization problem is non-convex and is solved by gradient descent or its variants. Unfortunately, little is known about global convergence properties of these approaches. Here we consider the problem of learning a concave function $f$ on a compact convex domain $Ω\subseteq {\mathbb R}^d$, using linear combinations of `bump-like' components (neurons). The parameters to be fitted are the centers of $N$ bumps, and the resulting empirical risk minimization problem is highly non-convex. We prove that, in the limit in which the number of neurons diverges, the evolution of gradient descent converges to a Wasserstein gradient flow in the space of probability distributions over $Ω$. Further, when the bump width $δ$ tends to $0$, this gradient flow has a limit which is a viscous porous medium equation. Remarkably, the cost function optimized by this gradient flow exhibits a special property known as displacement convexity, which implies exponential convergence rates for $N\to\infty$, $δ\to 0$. Surprisingly, this asymptotic theory appears to capture well the behavior for moderate values of $δ, N$. Explaining this phenomenon, and understanding the dependence on $δ,N$ in a quantitative manner remains an outstanding challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
