arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.09796stat.MLcs.LGmath.PR
收藏

核对齐风险估计:训练数据的风险预测

Kernel Alignment Risk Estimator: Risk Prediction from Training Data

Ecole Polytechnique Fédérale de Lausanne(洛桑联邦理工学院)

Arthur Jacot, Berfin Şimşek, Francesco Spadaro, Clément Hongler, Franck Gabriel

摘要:

我们研究了具有岭$λ>0$和i.i.d.的核$K$的核岭回归(KRR)的风险(即泛化误差)。意见。为此,我们引入了两个对象:信号捕获阈值(SCT)和核对齐风险估计(KARE)。SCT $\vartheta_{K,λ}$是数据分布的函数:它可用于识别KRR预测器捕获的数据的组成部分,并近似(预期)KRR风险。然后,这导致通过KARE $ρ_{K, λ}$的KRR风险近似,KARE是训练数据的显式函数,与真实数据分布无关。我们在函数环境中描述回归问题。关键的结果,然后从一般Wishart随机矩阵的Stieltjes变换的有限大小的分析。在一个自然的普遍性假设下(KRR矩渐近依赖于观测值的前两个矩),我们捕获KRR预测的均值和方差。我们数值研究我们的研究结果的希格斯和MNIST数据集的各种经典内核:KARE给出了一个很好的近似的风险,从而支持我们的普遍性假设。使用KARE,可以直接从训练集中比较内核和超参数的选择。因此,KARE提供了一个有前途的数据依赖过程来选择泛化良好的内核。

英文摘要:

We study the risk (i.e. generalization error) of Kernel Ridge Regression (KRR) for a kernel $K$ with ridge $λ>0$ and i.i.d. observations. For this, we introduce two objects: the Signal Capture Threshold (SCT) and the Kernel Alignment Risk Estimator (KARE). The SCT $\vartheta_{K,λ}$ is a function of the data distribution: it can be used to identify the components of the data that the KRR predictor captures, and to approximate the (expected) KRR risk. This then leads to a KRR risk approximation by the KARE $ρ_{K, λ}$, an explicit function of the training data, agnostic of the true data distribution. We phrase the regression problem in a functional setting. The key results then follow from a finite-size analysis of the Stieltjes transform of general Wishart random matrices. Under a natural universality assumption (that the KRR moments depend asymptotically on the first two moments of the observations) we capture the mean and variance of the KRR predictor. We numerically investigate our findings on the Higgs and MNIST datasets for various classical kernels: the KARE gives an excellent approximation of the risk, thus supporting our universality assumption. Using the KARE, one can compare choices of Kernels and hyperparameters directly from the training set. The KARE thus provides a promising data-dependent procedure to select Kernels that generalize well.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑