本文研究线性预测器良性过拟合的隐式偏差，证明在一般回归中最小范数插值通常不一致，而在分类中最大间隔预测器渐近偏向最小化加权平方 hinge 损失，从而为分类任务中的良性过拟合提供了新刻画。
摘要：
良性过拟合现象——即预测器完美拟合带噪训练数据的同时获得接近最优的期望损失——近年来受到广泛关注，但在良好设定的线性回归设定之外仍未得到充分理解。本文针对回归和分类任务，给出了关于何时可以或不可以期望良性过拟合发生的若干新结果。我们考虑一个用于线性预测器良性过拟合的原型且相当通用的数据模型，其中某个固定维度 $k$ 的任意输入分布与一个高维分布拼接。对于不一定良好设定的线性回归，我们证明最小范数插值预测器（标准训练方法收敛到该预测器）通常偏向于一个不一致的解，因此良性过拟合通常不会发生。此外，我们展示如何将其扩展到标准线性回归之外，通过一个论证证明：良性过拟合在某些回归问题上的存在性排除了它在其他回归问题上的存在性。随后我们转向分类问题，并证明那里的情况要有利得多。具体而言，我们证明最大间隔预测器（已知标准训练方法在方向上收敛到该预测器）在渐近意义上偏向于最小化一个加权的平方 hinge 损失。这使我们能够将分类中的良性过拟合问题归约为一个更简单的问题，即该损失是否是误分类误差的良好替代，并利用它在一些新设定中证明良性过拟合。
英文摘要：
The phenomenon of benign overfitting, where a predictor perfectly fits noisy training data while attaining near-optimal expected loss, has received much attention in recent years, but still remains not fully understood beyond well-specified linear regression setups. In this paper, we provide several new results on when one can or cannot expect benign overfitting to occur, for both regression and classification tasks. We consider a prototypical and rather generic data model for benign overfitting of linear predictors, where an arbitrary input distribution of some fixed dimension $k$ is concatenated with a high-dimensional distribution. For linear regression which is not necessarily well-specified, we show that the minimum-norm interpolating predictor (that standard training methods converge to) is biased towards an inconsistent solution in general, hence benign overfitting will generally not occur. Moreover, we show how this can be extended beyond standard linear regression, by an argument proving how the existence of benign overfitting on some regression problems precludes its existence on other regression problems. We then turn to classification problems, and show that the situation there is much more favorable. Specifically, we prove that the max-margin predictor (to which standard training methods are known to converge in direction) is asymptotically biased towards minimizing a weighted \emph{squared hinge loss}. This allows us to reduce the question of benign overfitting in classification to the simpler question of whether this loss is a good surrogate for the misclassification error, and use it to show benign overfitting in some new settings.
Understanding Generalization in Transformers: Error Bounds and Training Dynamics Under Benign and Harmful Overfitting
Zhongtai Securities Institute for Financial Studies, Shandong University（山东大学中泰证券金融研究院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）
Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions
Department of Statistics and Data Science, National University of Singapore（新加坡国立大学统计与数据科学系）；Department of Statistics and Data Science, Tsinghua University（清华大学统计与数据科学系）
Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression
The University of Tokyo（东京大学）；CyberAgent, Inc.（CyberAgent公司）
我们研究了良性过拟合理论在预测条件平均处理效应（CATE）中的应用，并建立了线性回归模型。随着因果推断的机器学习的发展，大范围的因果关系模型越来越受到关注。一个问题是，已经提出了怀疑，大规模的模型是倾向于过度拟合的样本选择的观察，因此，大型模型可能不适合因果预测。在这项研究中，为了解决可疑问题，我们通过应用最近的良性过拟合理论（Bartlett et al.，2020年）。具体来说，我们认为样本的分布开关取决于分配规则，并研究预测CATE的线性模型，其尺寸发散到无穷大。我们专注于两种方法：T-学习器，其基于每个治疗组的单独构造的估计量之间的差异，以及逆概率权重（IPW）-学习器，其解决由倾向分数近似的另一回归问题。在这两种方法中，估计器由完全拟合样本的插值器组成。结果表明，T-学习者除了随机分配外，无法实现一致性，而IPW-学习者在倾向分数已知的情况下，将风险收敛到零。这种差异源于T学习器无法保持协方差的特征空间，这对于过参数化设置中的良性过拟合是必要的。我们的结果提供了新的见解因果推断方法的使用在overparameterizated设置，特别是，双稳健估计。
(Mis)Understanding Benign Overfitting in Equity Return Prediction
Carl H. Lindner College of Business, University of Cincinnati（辛辛那提大学卡尔·H·林德纳商学院）；Católica Lisbon School of Business and Economics, Universidade Católica Portuguesa（葡萄牙天主教大学里斯本工商经济学院）；Department of Statistics, Pennsylvania State University（宾夕法尼亚州立大学统计系）
Benign Overfitting in Adversarial Training for Vision Transformers
Division of CEMSE, King Abdullah University of Science and Technology（科廷大学科学与工程学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；Department of Computer Science and Engineering, State University of New York at Buffalo（纽约州立大学布法罗分校计算机科学与工程学院）；School of Computer Science, University of Auckland（奥克兰大学计算机科学学院）
Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs
Benign overfitting beyond prediction: The ordinary least squares interpolator
Department of Data Sciences & Operations, University of Southern California（数据科学与运营系，南加州大学）；Department of Statistics, University of California, Davis（统计学系，加州大学戴维斯分校）；Department of Statistics, University of California, Berkeley（统计学系，加州大学伯克利分校）；Google DeepMind（谷歌DeepMind）
A finite sample analysis of the benign overfitting phenomenon for ridge function estimation
University of Lyon 2（里昂第二大学）；University of Oxford（牛津大学）；National Physical Laboratory（英国国家物理实验室）；The Alan Turing Institute（艾伦·图灵研究所）；FEMTO-ST Institute（FEMTO-ST研究所）
