A Function-Space Stability Boundary for Generalization in Interpolating Learning Systems
Ronald Katende
备注:
Theorem 5.2 does not establish the required quantitative neighbouring-dataset instability from its cited benign-overfitting setting. The claimed necessity direction and sharp stability boundary therefore lack a complete proof
Modern learning systems often interpolate training data while still generalizing well, yet it remains unclear when algorithmic stability explains this behavior. We model training as a function-space trajectory and measure sensitivity to single-sample perturbations along this trajectory. We propose a contractive propagation condition and a stability certificate obtained by unrolling the resulting recursion. A small certificate implies stability-based generalization, while we also prove that there exist interpolating regimes with small risk where such contractive sensitivity cannot hold, showing that stability is not a universal explanation. Experiments confirm that certificate growth predicts generalization differences across optimizers, step sizes, and dataset perturbations. The framework therefore identifies regimes where stability explains generalization and where alternative mechanisms must account for success.
Empirical Risk Minimization in the Interpolating Regime with Application to Neural Network Learning
Institute for Mathematical Stochastics(数学随机过程研究所);Technical University of Braunschweig(布伦瑞克工业大学);Institute for Stochastics and Applications(随机过程与应用研究所);University of Stuttgart(斯图加特大学)
Deep Neural Nets with Interpolating Function as Output Activation
University of California, Los Angeles(加州大学洛杉矶分校);Hong Kong University of Science and Technology(香港科技大学);Duke University(杜克大学);Tsinghua University(清华大学)
Stability and Generalization of the Decentralized Stochastic Gradient Descent
College of Computer, National University of Defense Technology(国防科技大学计算机学院);Scientific Computing & Imaging Institute, University of Utah(犹他大学科学计算与成像研究所)
Manifold Mixup: Better Representations by Interpolating Hidden States
Aalto Univeristy(阿尔托大学);Montréal Institute for Learning Algorithms(蒙特利尔学习算法研究所);Sharif University of Technology(谢里夫理工大学);Facebook AI Research(Facebook人工智能研究院)
Stability and Generalization for Markov Chain Stochastic Gradient Methods
Liu Bie Ju Centre for Mathematical Sciences, City University of Hong Kong(香港城市大学刘别驹数学科学中心);School of Computer Science, University of Birmingham(伯明翰大学计算机学院);School of Mathematics and Statistics, University of Sydney(悉尼大学数学与统计学院);Department of Mathematics and Statistics, State University of New York at Albany(纽约州立大学奥尔巴尼分校数学与统计系)
On the Stability and Generalization of Triplet Learning
Huazhong Agricultural University(华中农业大学);Southern University of Science and Technology(南方科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Massachusetts Institute of Technology(麻省理工学院);Spark Wave(火花波公司);Google Research(谷歌研究院);University of Southern California(南加利福尼亚大学);Courant Institute(库朗研究所);Cornell University(康奈尔大学)
Stability and Generalization for Stochastic Recursive Momentum-based Algorithms for (Strongly-)Convex One to $K$-Level Stochastic Optimizations
Central South University(中南大学);Tulane University(杜兰大学);National University of Defense Technology(国防科技大学);Xi’an Jiaotong University(西安交通大学);Shanghai Jiao Tong University(上海交通大学)
本文基于算法稳定性分析 STORM、COVER 和 SVMR 在凸与强凸一至 K 层随机优化中的泛化性,建立稳定性与超额风险界,并揭示估计器方差、层级增加及批量大小的影响。
School of Computer Science(计算机学院);Wuhan University(武汉大学);The University of Sydney(悉尼大学);School of Cyber Science and Technology(网络空间科学与技术学院);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Nanyang Technological University(南洋理工大学)
On the generalization of learning algorithms that do not converge
Institute for Data, Systems and Society(数据、系统与社会研究所);Massachusetts Institute of Technology(麻省理工学院);Prescient Design(Prescient设计公司);Genentech, Roche(罗氏基因泰克)
Deeper Insights into Deep Graph Convolutional Networks: Stability and Generalization
Department of Mathematics, College of Mathematics and Informatics, South China Agricultural University, Guangzhou, China(数学系,数学与信息学学院,华南农业大学,广州,中国);Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University, Jinhua, China(智能教育技术与应用浙江省重点实验室,浙江师范大学,金华,中国);Department of Mathematics, City University of Hong Kong, Hong Kong, China(数学系,香港城市大学,香港,中国)
Stability, Complexity and Data-Dependent Worst-Case Generalization Bounds
INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France(法国国家信息与自动化技术研究所、国家科学研究中心、巴黎高等师范学院计算机系/巴黎社会科学实验室);Department of Computing, Imperial College London, United Kingdom(伦敦帝国理工学院计算机系、英国);School of Computation and Technology, Technical University of Munich, Germany(慕尼黑技术大学计算与技术学院、德国);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心、德国)
Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Inria(法国国家信息与自动化研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学);Boğaziçi University(博阿齐奇大学);Rutgers University(罗格斯大学);Florida State University(佛罗里达州立大学);CNRS(法国国家科学研究中心)
最近的研究表明,重尾可以出现在随机最优化和尾部的沉重有联系的泛化错误。虽然这些研究揭示了现代环境中泛化行为的有趣方面,但它们依赖于强拓扑和统计规律性假设,这些假设很难在实践中得到验证。此外,经验表明,重尾和泛化之间的关系在实践中可能并不总是单调的,与现有理论的结论相反。在这项研究中,我们建立了新的尾部行为和随机梯度下降(SGD)的泛化性能之间的联系,通过镜头的算法稳定性。我们考虑一个二次优化问题,并使用一个重尾随机微分方程(及其欧拉离散化)作为一个代理建模的重尾行为出现在SGD。然后,我们证明了一致的稳定性界限,这揭示了以下结果:(i)没有任何外来的假设,我们表明,SGD将不会是稳定的,如果稳定性是衡量平方损失$x\mapsto x^2$,而它反过来成为稳定的,如果稳定性,而不是衡量与一些$p<2$的替代损失$x\mapsto |x|^p$。(ii)根据数据的方差,存在\emph{`threshold of heavy-tailedness'},使得泛化误差随着尾部变重而减小,只要尾部比该阈值轻。这表明,重尾和泛化之间的关系不是全局单调的。(iii)我们证明了匹配的一致稳定的下界,这意味着我们的界限是紧的沉重的尾巴。我们支持我们的理论与合成和真实的神经网络实验。