一致收敛可能无法解释深度学习中的泛化
Uniform convergence may be unable to explain generalization in deep learning
摘要：
为了解释超参数化深度网络令人惊讶的良好泛化行为，最近的工作已经为深度学习开发了各种泛化范围，所有这些都基于一致收敛的基本学习理论技术。虽然众所周知，这些现有的边界中有许多在数值上很大，但通过大量的实验，我们揭示了这些边界的一个更令人关注的方面：在实践中，这些边界可以随着训练数据集的大小{ \em增加}。在我们的观察的指导下，我们提出了过参数化线性分类器和梯度下降（GD）训练的神经网络的例子，其中一致收敛可证明不能“解释泛化”-即使我们考虑到GD { \em尽可能大的范围}的隐式偏差。更准确地说，即使我们只考虑GD输出的分类器集，在我们的设置中，这些分类器的测试误差小于一些小的$ε$，我们表明，在这组分类器上应用（双侧）一致收敛只会产生大于$1-ε$的空洞的泛化保证。通过这些发现，我们对基于一致收敛的泛化界限的能力表示怀疑，以提供过度参数化深度网络泛化良好的完整解释。
英文摘要：
Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bounds are numerically large, through numerous experiments, we bring to light a more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the training dataset size. Guided by our observations, we then present examples of overparameterized linear classifiers and neural networks trained by gradient descent (GD) where uniform convergence provably cannot "explain generalization" -- even if we take into account the implicit bias of GD {\em to the fullest extent possible}. More precisely, even if we consider only the set of classifiers output by GD, which have test errors less than some small $ε$ in our settings, we show that applying (two-sided) uniform convergence on this set of classifiers will yield only a vacuous generalization guarantee larger than $1-ε$. Through these findings, we cast doubt on the power of uniform convergence-based generalization bounds to provide a complete picture of why overparameterized deep networks generalize well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
