梯度方法在可分离数据上从不过拟合
Gradient Methods Never Overfit On Separable Data
摘要:
最近的一系列工作表明,当在可分离数据上训练线性预测器时,使用梯度方法和指数尾损失,预测器在最大边缘预测器的方向上渐近收敛。因此,预测变量不会渐近过拟合。然而,这并没有解决过拟合是否会在一定数量的迭代之后非渐进地发生的问题。在本文中,我们正式表明,标准梯度方法,(特别是梯度流、梯度下降和随机梯度下降)在可分离数据上永远不会过拟合:如果我们在大小为$m$的数据集上运行这些方法进行$T$迭代,则经验风险和泛化误差都以$\tilde{\mathcal{O}}(1/γ^2 T)$的基本最优速率下降,直到$T\approx m$,在这一点上,泛化误差保持固定在$\tilde{\mathcal{O}}(1/γ^2 m)$的基本最优水平,而不管$T$有多大。在此过程中,我们提出了数据集上违反保证金的数量的非渐近界,并证明了它们的紧密性。
英文摘要:
A line of recent works established that when training linear predictors over separable data, using gradient methods and exponentially-tailed losses, the predictors asymptotically converge in direction to the max-margin predictor. As a consequence, the predictors asymptotically do not overfit. However, this does not address the question of whether overfitting might occur non-asymptotically, after some bounded number of iterations. In this paper, we formally show that standard gradient methods (in particular, gradient flow, gradient descent and stochastic gradient descent) never overfit on separable data: If we run these methods for $T$ iterations on a dataset of size $m$, both the empirical risk and the generalization error decrease at an essentially optimal rate of $\tilde{\mathcal{O}}(1/γ^2 T)$ up till $T\approx m$, at which point the generalization error remains fixed at an essentially optimal level of $\tilde{\mathcal{O}}(1/γ^2 m)$ regardless of how large $T$ is. Along the way, we present non-asymptotic bounds on the number of margin violations over the dataset, and prove their tightness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。