On the Dynamics of Delayed Generalization in Linear Models
Department of XXX, University of YYY, Location, Country（YYY大学XXX系）；School of ZZZ, Institute of WWW, Location, Country（WWW研究所ZZZ学院）；Indian Institute of Technology Madras, Chennai（马德拉斯印度理工学院）
Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.
Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent
Intelligent Game and Decision Lab（智能游戏与决策实验室）；College of Computer Science and Technology（计算机科学与技术学院）；National University of Defense Technology（国防科技大学）；Sun Yat-sen University（中山大学）；JD Explore Academy（京东探索研究院）
