Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regime
Center for Machine Intelligence and Data Science Indian Institute of Technology Bombay(机器智能与数据科学中心 印度理工学院班加罗尔);International Computer Science Institute Department of Statistics University of California, Berkeley(国际计算机科学研究所 统计学系 加州大学伯克利分校);Department of Mathematics University of Southern California(数学系 华盛顿大学)
Kernel ridge regression (KRR) is a popular class of machine learning models that has become an important tool for understanding deep learning. Much of the focus thus far has been on studying the proportional asymptotic regime, $n \asymp d$, where $n$ is the number of training samples and $d$ is the dimension of the dataset. In the proportional regime, under certain conditions on the data distribution, the kernel random matrix involved in KRR exhibits behavior akin to that of a linear kernel. In this work, we extend the study of kernel regression to the quadratic asymptotic regime, where $n \asymp d^2$. In this regime, we demonstrate that a broad class of inner-product kernels exhibits behavior similar to a quadratic kernel. Specifically, we establish an operator norm approximation bound for the difference between the original kernel random matrix and a quadratic kernel random matrix with additional correction terms compared to the Taylor expansion of the kernel functions. The approximation works for general data distributions under a Gaussian-moment-matching assumption with a covariance structure. This new approximation is utilized to obtain a limiting spectral distribution of the original kernel matrix and characterize the precise asymptotic training and test errors for KRR in the quadratic regime when $n/d^2$ converges to a non-zero constant. The generalization errors are obtained for (i) a random teacher model, (ii) a deterministic teacher model where the weights are perfectly aligned with the covariance of the data. Under the random teacher model setting, we also verify that the generalized cross-validation (GCV) estimator can consistently estimate the generalization error in the quadratic regime for anisotropic data. Our proof techniques combine moment methods, Wick's formula, orthogonal polynomials, and resolvent analysis of random matrices with correlated entries.
Maximal Margin Distribution Support Vector Regression with coupled Constraints-based Convex Optimization
Shanghai Tenth People's Hospital(上海市第十人民医院);Tongji University(同济大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Jilin University(吉林大学);The Ohio State University(俄亥俄州立大学)
Deterministic equivalent and error universality of deep random features learning
ETH Zurich(苏黎世联邦理工学院);ETH AI Center(苏黎世联邦理工学院人工智能中心);École Normale Supérieure (ENS) - PSL(巴黎高等师范学院(巴黎文理研究大学));CNRS(法国国家科学研究中心);École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)