梯度下降与投影在学习低次球面多项式中的应用：近最优最小最大学习率
Gradient Descent with Projection Finds Over-Parameterized Neural Networks for Learning Low-Degree Polynomials with Nearly Minimax Optimal Rate
AI总结：
本文研究了通过训练过参数化的两层神经网络学习低次球面多项式的问题，提出了一种改进的梯度下降与投影算法，实现了近最优的学习率，且在未知真实次数时能自适应选择最优参数。
摘要：
我们研究了通过训练一个过参数化的两层神经网络来学习定义在$\RR^d$单位球上的低次球面多项式的问题。我们的主要结果是学习此类低次多项式显著改进的样本复杂度。我们证明，对于任何回归风险$\eps \in (0, Θ(d^{-k_0})]$，一个通过新型梯度下降与投影（GDP）训练的过参数化两层神经网络需要样本复杂度为$n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)$，以概率$1-δ$，在$δ\in (0,1)$的情况下，与代表性的样本复杂度$Θ(d^{k_0} \max\set{\eps^{-2},\log d})$相比有显著改进。此外，这种样本复杂度几乎无法进一步改进，因为训练后的网络实现了近最优的非参数回归风险，其顺序为$\log({4}/δ) \cdot Θ(d^{k_0}/{n})$，以至少$1-δ$的概率。另一方面，具有秩$Θ(d^{k_0})$的核的最小最大回归风险为$Θ(d^{k_0}/{n})$，因此通过GDP训练的网络的非参数回归风险速率几乎是最小最大最优的。在真实次数$k_0$未知的情况下，我们提出了一种新的且可证明的自适应次数选择算法，能够识别真实次数并实现相同的近最优回归速率。据我们所知，这是首次通过训练一个过参数化的神经网络（使用流行的激活函数ReLU）并具有算法保证来学习低次球面多项式时获得近最优风险界。由于GDP的特征学习能力，我们的结果超出了常规神经切线核（NTK）的限制。
英文摘要：
We study the problem of learning a low-degree spherical polynomial of degree $k_0 = Θ(1) \ge 1$ defined on the unit sphere in $\RR^d$ by training an over-parameterized two-layer neural network with augmented feature in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk $\eps \in (0, Θ(d^{-k_0})]$, an over-parameterized two-layer neural network trained by a novel Gradient Descent with Projection (GDP) requires a sample complexity of $n \asymp Θ( \log(4/δ) \cdot d^{k_0}/\eps)$ with probability $1-δ$ for $δ\in (0,1)$, in contrast with the representative sample complexity $Θ(d^{k_0} \max\set{\eps^{-2},\log d})$. Moreover, such sample complexity is nearly unimprovable since the trained network renders a nearly optimal rate of the nonparametric regression risk of the order $\log({4}/δ) \cdot Θ(d^{k_0}/{n})$ with probability at least $1-δ$. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank $Θ(d^{k_0})$ is $Θ(d^{k_0}/{n})$, so that the rate of the nonparametric regression risk of the network trained by GDP is nearly minimax optimal. In the case that the ground truth degree $k_0$ is unknown, we present a novel and provable adaptive degree selection algorithm which identifies the true degree and achieves the same nearly optimal regression rate. To the best of our knowledge, this is the first time that a nearly optimal risk bound is obtained by training an over-parameterized neural network with a popular activation function (ReLU) and algorithmic guarantee for learning low-degree spherical polynomials. Due to the feature learning capability of GDP, our results are beyond the regular Neural Tangent Kernel (NTK) limit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
