Transfer learning is a critical part of real-world machine learning deployments and has been extensively studied in experimental works with overparameterized neural networks. However, even in the simplest setting of linear regression a notable gap still exists in the theoretical understanding of transfer learning. In-distribution research on high-dimensional linear regression has led to the identification of a phenomenon known as \textit{benign overfitting}, in which linear interpolators overfit to noisy training labels and yet still generalize well. This behavior occurs under specific conditions on the source covariance matrix and input data dimension. Therefore, it is natural to wonder how such high-dimensional linear models behave under transfer learning. We prove the first non-asymptotic excess risk bounds for benignly-overfit linear interpolators in the transfer learning setting. From our analysis, we propose a taxonomy of \textit{beneficial} and \textit{malignant} covariate shifts based on the degree of overparameterization. We follow our analysis with empirical studies that show these beneficial and malignant covariate shifts for linear interpolators on real image data, and for fully-connected neural networks in settings where the input data dimension is larger than the training sample size.
Convexifying Sparse Interpolation with Infinitely Wide Neural Networks: An Atomic Norm Approach
University of Minnesota（明尼苏达大学）
这项工作探讨了通过稀疏（神经元计数），无限宽，单隐层神经网络泄漏整流线性单元激活的精确数据插值的问题。使用[Rasekaran et al.，2012]中，我们在对网络权值和偏置的几种不同约束下，推导出了该问题相应原子集的凸包的简单刻画，从而得到了这些问题的等价凸公式。一个适度的扩展，我们提出的框架，二进制分类问题。我们通过实验探索了由此产生的配方的有效性，并与通过梯度下降训练的网络进行了比较。
Generalization error of min-norm interpolators in transfer learning
Department of Statistics, Harvard University（哈佛大学统计系）；Department of Statistics, Stanford University（斯坦福大学统计系）；Department of Statistics, University of Florida（佛罗里达大学统计系）
