Multilayer perceptrons (MLP) with one hidden layer have been used for a long time to deal with non-linear regression. However, in some task, MLP's are too powerful models and a small mean square error (MSE) may be more due to overfitting than to actual modelling. If the noise of the regression model is Gaussian, the overfitting of the model is totally determined by the behavior of the likelihood ratio test statistic (LRTS), however in numerous cases the assumption of normality of the noise is arbitrary if not false. In this paper, we present an universal bound for the overfitting of such model under weak assumptions, this bound is valid without Gaussian or identifiability assumptions. The main application of this bound is to give a hint about determining the true architecture of the MLP model when the number of data goes to infinite. As an illustration, we use this theoretical result to propose and compare effective criteria to find the true architecture of an MLP.
Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression
The University of Tokyo（东京大学）；CyberAgent, Inc.（CyberAgent公司）
我们研究了良性过拟合理论在预测条件平均处理效应（CATE）中的应用，并建立了线性回归模型。随着因果推断的机器学习的发展，大范围的因果关系模型越来越受到关注。一个问题是，已经提出了怀疑，大规模的模型是倾向于过度拟合的样本选择的观察，因此，大型模型可能不适合因果预测。在这项研究中，为了解决可疑问题，我们通过应用最近的良性过拟合理论（Bartlett et al.，2020年）。具体来说，我们认为样本的分布开关取决于分配规则，并研究预测CATE的线性模型，其尺寸发散到无穷大。我们专注于两种方法：T-学习器，其基于每个治疗组的单独构造的估计量之间的差异，以及逆概率权重（IPW）-学习器，其解决由倾向分数近似的另一回归问题。在这两种方法中，估计器由完全拟合样本的插值器组成。结果表明，T-学习者除了随机分配外，无法实现一致性，而IPW-学习者在倾向分数已知的情况下，将风险收敛到零。这种差异源于T学习器无法保持协方差的特征空间，这对于过参数化设置中的良性过拟合是必要的。我们的结果提供了新的见解因果推断方法的使用在overparameterizated设置，特别是，双稳健估计。
Analysis of overfitting in the regularized Cox model
Institute for Mathematical and Molecular Biomedicine, King’s College London（伦敦国王学院数学与分子生物医学研究所）；King’s College London（伦敦国王学院）；Saddle Point Science Ltd（鞍点科学有限公司）
Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；School of Mathematical Sciences（数学科学学院）；Peking University（北京大学）；Department of Computer Science（计算机科学系）；University of Warwick（沃里克大学）；Industrial Engineering & Management Sciences（工业工程与管理科学）；Northwestern University（西北大学）
