We study the generalization capability of nearly-interpolating linear regressors: $\boldsymbolβ$'s whose training error $τ$ is positive but small, i.e., below the noise floor. Under a random matrix theoretic assumption on the data distribution and an eigendecay assumption on the data covariance matrix $\boldsymbolΣ$, we demonstrate that any near-interpolator exhibits rapid norm growth: for $τ$ fixed, $\boldsymbolβ$ has squared $\ell_2$-norm $\mathbb{E}[\|{\boldsymbolβ}\|_{2}^{2}] = Ω(n^α)$ where $n$ is the number of samples and $α>1$ is the exponent of the eigendecay, i.e., $λ_i(\boldsymbolΣ) \sim i^{-α}$. This implies that existing data-independent norm-based bounds are necessarily loose. On the other hand, in the same regime we precisely characterize the asymptotic trade-off between interpolation and generalization. Our characterization reveals that larger norm scaling exponents $α$ correspond to worse trade-offs between interpolation and generalization. We verify empirically that a similar phenomenon holds for nearly-interpolating shallow neural networks.
Generalization error of min-norm interpolators in transfer learning
Department of Statistics, Harvard University（哈佛大学统计系）；Department of Statistics, Stanford University（斯坦福大学统计系）；Department of Statistics, University of Florida（佛罗里达大学统计系）
A kinetic-based regularization method for data science applications
Engineering Mechanics Unit, Jawaharlal Nehru Centre for Advanced Scientific Research（贾瓦哈拉尔·奈尔中心先进科学研究中心工程力学单元）；CCS-2 Computational Physics and Methods（计算物理学与方法CCS-2）；Center for Nonlinear Studies (CNLS)（非线性研究中心（CNLS））；Robotics Technology Park, Indian Institute of Science（机器人技术公园，印度科学研究院）；Fondazione Instituto Italiano di Tecnologia（意大利技术基金会）
