发表机构
Johns Hopkins University; University of California San Diego（约翰霍普金斯大学; 加利福尼亚大学圣迭戈分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出理论框架，分析带权重衰减的梯度下降训练神经网络的泛化动力学，通过误差分解与近似齐次性导出逐层界限，解释泛化差异并刻画grokking现象。
AI 中文摘要
理解泛化仍然是机器学习中的一个核心挑战，因为它需要同时考虑数据、架构和训练动力学。在本文中，我们开发了一个理论框架，用以刻画这些因素如何在训练过程中共同塑造泛化性能。更具体地，我们研究了一类在$\ell^2$损失下通过带权重衰减的梯度下降（GD）训练的广泛神经网络，并证明了GD收敛到经验损失全局极小值的一个邻域。通过基于输入数据对空间进行划分，我们将总体误差分解为数据误差、优化误差和预测变化误差，并分别对它们进行界定。特别是，对于衡量学习函数振荡的预测变化误差，我们提出了（局部）近似齐次性，并推导了其沿训练轨迹演化的显式逐单元和逐层界限。这些界限产生了两个重要推论：改进泛化的必要条件解释了逐层泛化行为的差异；一个充分条件描述了延迟泛化，并为grokking提供了理论刻画。
英文摘要
Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the $\ell^2$ loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.
