arXiv 2103.045542021-03-08cs.LGmath.STstat.MLstat.TH

Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models

University of California, Berkeley（加州大学伯克利分校）；Salesforce Research（Salesforce研究院）

Zitong Yang, Yu Bai, Song Mei

英文摘要：

Recent work showed that there could be a large gap between the classical uniform convergence bound and the actual test error of zero-training-error predictors (interpolators) such as deep neural networks. To better understand this gap, we study the uniform convergence in the nonlinear random feature model and perform a precise theoretical analysis on how uniform convergence depends on the sample size and the number of parameters. We derive and prove analytical expressions for three quantities in this model: 1) classical uniform convergence over norm balls, 2) uniform convergence over interpolators in the norm ball (recently proposed by Zhou et al. (2020)), and 3) the risk of minimum norm interpolator. We show that, in the setting where the classical uniform convergence bound is vacuous (diverges to $\infty$), uniform convergence over the interpolators still gives a non-trivial bound of the test error of interpolating solutions. We also showcase a different setting where classical uniform convergence bound is non-vacuous, but uniform convergence over interpolators can give an improved sample complexity guarantee. Our result provides a first exact comparison between the test errors and uniform convergence bounds for interpolators beyond simple linear models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

通过平滑激活函数缓解深度神经网络一致收敛中的维度灾难

Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations

Department of Statistics, The Pennsylvania State University（宾夕法尼亚州立大学统计学系）

本文通过分析平滑激活深度神经网络，建立了统一收敛的理论框架，证明其能够通过自适应利用目标函数的低维层次组合结构来缓解维度灾难。

2026-06-04cs.LGmath.STstat.ME

基于 Fat-Shattering 维数的改进一致收敛界

An Improved Uniform Convergence Bound with Fat-Shattering Dimension

Università degli Studi di Milano（米兰大学）；Istituto Italiano di Tecnologia（意大利理工学院）

本文针对实值函数一致收敛界中上下界因对数平方因子导致的差距，利用 fat-shattering 维数提出改进上界并闭合该差距。

2023-07-13cs.LGstat.ML

Uniform Generalization, Concentration, and Adaptive Learning

KAUST（阿卜杜拉国王科技大学）

One fundamental goal in any learning algorithm is to mitigate its risk for overfitting. Mathematically, this requires that the learning algorithm enjoys a small generalization risk, which is defined either in expectation or in probability. Both types of generalization are commonly used in the literature. For instance, generalization in expectation has been used to analyze algorithms, such as ridge regression and SGD, whereas generalization in probability is used in the VC theory, among others. Recently, a third notion of generalization has been studied, called uniform generalization, which req…

2016-10-03cs.ITcs.LGmath.IT

Uniform convergence may be unable to explain generalization in deep learning

Carnegie Mellon University（卡内基梅隆大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）

Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bounds are numerically large, through numerous experiments, we bring to light a more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the training dataset size. Guided by our observations, we then present examples of overparameterized linear class…

2021-10-17cs.LGstat.ML

高斯核岭回归的一致收敛性

Uniform convergence for Gaussian kernel ridge regression

本文首次证明固定超参数的高斯核岭回归在一致范数和L2范数下具有多项式收敛速率，填补理论空白，为非参数回归提供理论依据。

2025-09-11cs.LGstat.ML

On Uniform Convergence and Low-Norm Interpolation Learning

University of Chicago（芝加哥大学）；TTI-Chicago（芝加哥丰田技术研究院）

We consider an underdetermined noisy linear regression model where the minimum-norm interpolating predictor is known to be consistent, and ask: can uniform convergence in a norm ball, or at least (following Nagarajan and Kolter) the subset of a norm ball that the algorithm selects on a typical input set, explain this success? We show that uniformly bounding the difference between empirical and population errors cannot show any learning in the norm ball, and cannot show consistency for any set, even one depending on the exact algorithm and distribution. But we argue we can explain the consisten…

2021-01-14cs.LGstat.ML

更多相关论文收起更多论文

Submultiplicative Glivenko-Cantelli and Uniform Convergence of Revenues

Tel Aviv University（特拉维夫大学）；Microsoft Research（微软研究院）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Google Research（谷歌研究院）；Institute for Advanced Study（普林斯顿高等研究院）；Technion — Israel Institute of Technology（以色列理工学院）

In this work we derive a variant of the classic Glivenko-Cantelli Theorem, which asserts uniform convergence of the empirical Cumulative Distribution Function (CDF) to the CDF of the underlying distribution. Our variant allows for tighter convergence bounds for extreme values of the CDF. We apply our bound in the context of revenue learning, which is a well-studied problem in economics and algorithmic game theory. We derive sample-complexity bounds on the uniform convergence rate of the empirical revenues to the true revenues, assuming a bound on the $k$th moment of the valuations, for any (po…

2017-11-06cs.GTcs.LG

Bennett-type Generalization Bounds: Large-deviation Case and Faster Rate of Convergence

The Biodesign Institute（生物设计研究所）；Arizona State University（亚利桑那州立大学）

In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we first develop two types of Bennett-type deviation inequality for the i.i.d. learning process: one provides the generalization bounds based on the uniform entropy number; the other leads to the bounds based on the Rademacher complexity. We then adopt a new method to obtain the alternative expressions of the Bennett-type generalization bounds, which imply that the…

2013-09-26cs.LGstat.ML

Uniform Generalization Bound on Time and Inverse Temperature for Gradient Descent Algorithm and its Application to Analysis of Simulated Annealing

NEC Corporation（日本电气股份有限公司）；Biometrics Research Laboratories, NEC Corporation（日本电气股份有限公司生物识别研究实验室）

In this paper, we propose a novel uniform generalization bound on the time and inverse temperature for stochastic gradient Langevin dynamics (SGLD) in a non-convex setting. While previous works derive their generalization bounds by uniform stability, we use Rademacher complexity to make our generalization bound independent of the time and inverse temperature. Using Rademacher complexity, we can reduce the problem to derive a generalization bound on the whole space to that on a bounded region and therefore can remove the effect of the time and inverse temperature from our generalization bound. …

2022-06-05cs.LGmath.PRstat.ML

Generalization Bounds for Uniformly Stable Algorithms

Google Brain（谷歌大脑）；Stanford University（斯坦福大学）

Uniform stability of a learning algorithm is a classical notion of algorithmic stability introduced to derive high-probability bounds on the generalization error (Bousquet and Elisseeff, 2002). Specifically, for a loss function with range bounded in $[0,1]$, the generalization error of a $γ$-uniformly stable learning algorithm on $n$ samples is known to be within $O((γ+1/n) \sqrt{n \log(1/δ)})$ of the empirical error with probability at least $1-δ$. Unfortunately, this bound does not lead to meaningful generalization bounds in many common settings where $γ\geq 1/\sqrt{n}$. At the same time the…

2019-03-18cs.DScs.LGstat.ML

An Exploration of Multicalibration Uniform Convergence Bounds

University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）

Recent works have investigated the sample complexity necessary for fair machine learning. The most advanced of such sample complexity bounds are developed by analyzing multicalibration uniform convergence for a given predictor class. We present a framework which yields multicalibration error uniform convergence bounds by reparametrizing sample complexities for Empirical Risk Minimization (ERM) learning. From this framework, we demonstrate that multicalibration error exhibits dependence on the classifier architecture as well as the underlying data distribution. We perform an experimental evalua…

2022-02-09cs.LG

AdaL: Adaptive Gradient Transformation Contributes to Convergences and Generalizations

Beijing Institute of Technology（北京理工大学）；Shenzhen University（深圳大学）

Adaptive optimization methods have been widely used in deep learning. They scale the learning rates adaptively according to the past gradient, which has been shown to be effective to accelerate the convergence. However, they suffer from poor generalization performance compared with SGD. Recent studies point that smoothing exponential gradient noise leads to generalization degeneration phenomenon. Inspired by this, we propose AdaL, with a transformation on the original gradient. AdaL accelerates the convergence by amplifying the gradient in the early stage, as well as dampens the oscillation an…

2021-07-04cs.AIcs.LG

平方根Lipschitz损失下的均匀收敛

Uniform Convergence with Square-Root Lipschitz Loss

University of Chicago（芝加哥大学）；Stanford University（斯坦福大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）

本文针对高斯数据，利用Rademacher复杂度和损失平方根的Lipschitz常数，建立了更广泛的均匀收敛保证，推广了基于光滑性的结果，并适用于相位恢复和ReLU回归等非光滑损失场景。

2023-06-22cs.LGstat.ML

On Convergence and Generalization of Dropout Training

Johns Hopkins University（约翰斯·霍普金斯大学）；Department of Computer Science（计算机科学系）

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations. Under mild overparametrization and assuming that the limiting kernel can separate the data distribution with a positive margin, we show that dropout training with logistic loss achieves $ε$-suboptimality in test error in $O(1/ε)$ iterations.

2020-10-23cs.LGstat.ML

具有 Lipschitz 连续激活函数和可变宽度的深度神经网络的一致收敛性

Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths

Old Dominion University（奥尔多明尼恩大学）；Sun Yat-sen University（中山大学）

本文研究可变宽度深度神经网络在层数趋于无穷时的一致收敛性，给出权重、偏置与 Lipschitz 常数的充分条件，并推广至固定、有界、无界宽度及卷积神经网络。

2023-06-02cs.LG

Uniform Convergence Bounds for Codec Selection

We frame the problem of selecting an optimal audio encoding scheme as a supervised learning task. Through uniform convergence theory, we guarantee approximately optimal codec selection while controlling for selection bias. We present rigorous statistical guarantees for the codec selection problem that hold for arbitrary distributions over audio sequences and for arbitrary quality metrics. Our techniques can thus balance sound quality and compression ratio, and use audio samples from the distribution to select a codec that performs well on that particular type of data. The applications of our t…

2018-12-18cs.LGcs.SDeess.AS