arXiv 2210.085712022-10math.STstat.MLstat.TH

无维度岭回归

Dimension free ridge regression

Chen Cheng, Andrea Montanari

备注：: 86 pages, 3 figures. Updated to the published journal version
发表信息：: The Annals of Statistics 52.6 (2024): 2879-2912

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出无维度岭回归理论，超越比例渐近，在希尔伯特空间特征下建立非渐近近似界，刻画正则变化谱和良性过拟合。

摘要：

随机矩阵理论已成为高维统计和理论机器学习中广泛使用的工具。然而，随机矩阵理论主要关注比例渐近，即数据矩阵的列数随行数成比例增长。在统计学中，列对应协变量，行对应样本，这并不总是最自然的设定。为了超越比例渐近，我们重新审视独立同分布数据$(x_i, y_i)$, $i\le n$上的岭回归（$\ell_2$惩罚最小二乘），其中$x_i$是特征向量，$y_i = β^\top x_i +ε_i \in\mathbb{R}$是响应。我们允许特征向量是高维的，甚至无限维的，此时它属于可分希尔伯特空间，并假设$z_i := Σ^{-1/2}x_i$具有独立同分布的元素，或满足某种凸集中性质。在此设定下，我们建立了非渐近界，用“等价”序列模型（具有对角设计矩阵的回归模型）的偏差和方差来近似岭回归的偏差和方差。近似误差由因子$(1\pm Δ)$控制，其中$Δ$显式地小。此前，这种近似结果仅在比例机制下且仅达到加性误差时已知：特别是，它无法刻画当超额风险收敛到0时的行为。我们的通用理论恢复了比例机制下的早期结果（具有更好的误差率）。作为新的应用，我们获得了具有正则变化谱的希尔伯特协变量岭回归的完全显式和尖锐刻画。最后，我们分析了过参数化近插值设定，并获得了尖锐的“良性过拟合”保证。

英文摘要：

Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression ($\ell_2$-penalized least squares) on i.i.d. data $(x_i, y_i)$, $i\le n$, where $x_i$ is a feature vector and $y_i = β^\top x_i +ε_i \in\mathbb{R}$ is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either $z_i := Σ^{-1/2}x_i$ to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish non-asymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an `equivalent' sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by $(1\pm Δ)$ for some explicitly small $Δ$. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to $0$. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp `benign overfitting' guarantees.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文