arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01062stat.MLcs.LGmath.PRmath.STstat.TH
收藏

二次情形下核随机矩阵与核回归的普适性

Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regime

Center for Machine Intelligence and Data Science Indian Institute of Technology Bombay(机器智能与数据科学中心 印度理工学院班加罗尔);International Computer Science Institute Department of Statistics University of California, Berkeley(国际计算机科学研究所 统计学系 加州大学伯克利分校);Department of Mathematics University of Southern California(数学系 华盛顿大学)

Parthe Pandit, Zhichao Wang, Yizhe Zhu

AI总结:

本文将核回归研究拓展至$n \asymp d^2$的二次渐近情形,证明内积核表现出二次核行为,推导了核矩阵极限谱分布与KRR渐近误差,验证了GCV估计的一致性。

摘要:

核岭回归(KRR)是一类广泛应用的机器学习模型,已成为理解深度学习的重要工具。目前的研究大多聚焦于比例渐近情形($n \asymp d$),其中$n$为训练样本数,$d$为数据集维度。在该比例情形下,当数据分布满足特定条件时,KRR涉及的核随机矩阵会表现出与线性核类似的特性。\n本文将核回归的研究拓展至二次渐近情形,即$n \asymp d^2$的场景。我们证明,在该情形下,一大类内积核会表现出与二次核相似的行为。具体而言,我们建立了原始核随机矩阵与二次核随机矩阵差值的算子范数近似界,与核函数的泰勒展开相比,该近似额外引入了校正项。该近似在数据分布满足带协方差结构的高斯矩匹配假设下成立,适用于一般数据分布。\n利用这一新的近似结果,我们得到了原始核矩阵的极限谱分布,并刻画了当$n/d^2$收敛于非零常数时,二次情形下KRR精确的渐近训练误差与测试误差。我们分别在两种场景下推导了泛化误差:(i)随机教师模型;(ii)权重与数据协方差完全对齐的确定性教师模型。在随机教师模型设定下,我们还验证了广义交叉验证(GCV)估计量在二次情形下能够对各向异性数据的泛化误差进行一致估计。我们的证明技术结合了矩方法、Wick公式、正交多项式,以及带相关元素的随机矩阵的预解式分析。

英文摘要:

Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus thus far has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In the proportional regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibits behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and test errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for (i) a random teacher model, (ii) a deterministic teacher model where the weights are perfectly aligned with the covariance of the data. Under the random teacher model setting, we also verify that the generalized cross-validation (GCV) estimator can consistently estimate the generalization error in the quadratic regime for anisotropic data. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑