高维线性化双层神经网络
Linearized two-layers neural networks in high dimension
摘要:
我们考虑学习一个未知的功能$f_{\star}$的$d$维球体上的平方损失,给定独立同分布的问题。采样$\{(y_i,{\boldsymbol x}_i)\}_{i\le n}$和$y_i=f_{\star}({\boldsymbol x}_i)+\varepsilon_i$,其中${\boldsymbol x}_i$是均匀分布在球体上的特征向量。我们研究了两类流行的模型,它们可以被视为两层神经网络围绕随机初始化的线性化:Rahimi-Recht(RF)的随机特征模型; Jacot-Gabriel-Hongler(NT)的神经切线核模型。这两种方法也可以被视为随机近似的内核岭回归(相对于不同的内核),并享有普遍的近似性能时,神经元$N$发散,为一个固定的尺寸$d$。我们考虑两个具体的制度:近似限制制度,其中$n=\infty$,而$d$和$N$是大的,但有限的;和样本量有限的制度,其中$N=\infty$,而$d$和$n$是大的,但有限的。在第一个制度中,我们证明,如果$d^{\ell + δ} \le N\le d^{\ell+1-δ}$小$δ> 0$,那么\RF \,有效地适合度$\ell$多项式的原始功能,和\NT \,适合度$(\ell+1)$多项式。在第二种情况下,RF和NT都简化为具有旋转不变核的核方法。我们证明了,如果样本的数量是$d^{\ell + δ} \le n \le d^{\ell +1-δ}$,那么核方法最多可以适合一个程度$\ell$多项式的原始功能。这个下界是通过核岭回归实现的。最佳的预测误差为消失岭正则化。
英文摘要:
We consider the problem of learning an unknown function $f_{\star}$ on the $d$-dimensional sphere with respect to the square loss, given i.i.d. samples $\{(y_i,{\boldsymbol x}_i)\}_{i\le n}$ where ${\boldsymbol x}_i$ is a feature vector uniformly distributed on the sphere and $y_i=f_{\star}({\boldsymbol x}_i)+\varepsilon_i$. We study two popular classes of models that can be regarded as linearizations of two-layers neural networks around a random initialization: the random features model of Rahimi-Recht (RF); the neural tangent kernel model of Jacot-Gabriel-Hongler (NT). Both these approaches can also be regarded as randomized approximations of kernel ridge regression (with respect to different kernels), and enjoy universal approximation properties when the number of neurons $N$ diverges, for a fixed dimension $d$. We consider two specific regimes: the approximation-limited regime, in which $n=\infty$ while $d$ and $N$ are large but finite; and the sample size-limited regime in which $N=\infty$ while $d$ and $n$ are large but finite. In the first regime we prove that if $d^{\ell + δ} \le N\le d^{\ell+1-δ}$ for small $δ> 0$, then \RF\, effectively fits a degree-$\ell$ polynomial in the raw features, and \NT\, fits a degree-$(\ell+1)$ polynomial. In the second regime, both RF and NT reduce to kernel methods with rotationally invariant kernels. We prove that, if the number of samples is $d^{\ell + δ} \le n \le d^{\ell +1-δ}$, then kernel methods can fit at most a a degree-$\ell$ polynomial in the raw features. This lower bound is achieved by kernel ridge regression. Optimal prediction error is achieved for vanishing ridge regularization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。