We show that kernel-based quadrature rules for computing integrals can be seen as a special case of random feature expansions for positive definite kernels, for a particular decomposition that always exists for such kernels. We provide a theoretical analysis of the number of required samples for a given approximation error, leading to both upper and lower bounds that are based solely on the eigenvalues of the associated integral operator and match up to logarithmic terms. In particular, we show that the upper bound may be obtained from independent and identically distributed samples from a specific non-uniform distribution, while the lower bound if valid for any set of points. Applying our results to kernel-based quadrature, while our results are fairly general, we recover known upper and lower bounds for the special cases of Sobolev spaces. Moreover, our results extend to the more general problem of full function approximations (beyond simply computing an integral), with results in L2- and L$\infty$-norm that match known results for special cases. Applying our results to random features, we show an improvement of the number of random features needed to preserve the generalization guarantees for learning with Lipschitz-continuous losses.
Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models
RIKEN AIP(日本理化学研究所革新智能综合研究中心)
无限宽模型是特征图的加权积分$\int φ(x,v) d μ(v)$。该模型擅长处理无限数量的特征,因此已被用于深度学习的理论研究。核求积是一种基于核的数值积分方案,用于快速逼近期望值$\int f(x) d p(x)$。在这项研究中,关于重量$μ$作为一个有符号(或复杂/向量值)分布的参数,我们开发的一般核求积(GKQ)的参数分布。所提出的方法可以实现一个快速的近似速度$O(e^{-p})$与参数数$p$,这是比传统的巴伦的速度,和一个快速的估计速度$\widetilde{O}(1/n)$与样本量$n$。因此,我们得到了一个新的基于范数的复杂性度量的无限宽模型。由于GKQ隐式地进行经验风险最小化,因此我们可以理解复杂性度量也反映了梯度学习设置中的泛化性能。
University of Warwick, Department of Statistics(沃里克大学统计系);Imperial College London, Department of Mathematics(伦敦帝国理工学院数学系);Newcastle University, School of Mathematics(新castle大学数学系);University of Technology Sydney, School of Mathematical(悉尼技术大学数学与物理科学学院);The Alan Turing Institute for Data Science(数据科学阿尔图里研究所)
Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature
Lattice Lab, Toyota Motor Corporation / Machine Learning Research Group, University of Oxford(丰田公司 lattice 实验室 / 机器学习研究组,牛津大学);The University of Osaka / Lattice Lab, Toyota Motor Corporation / RIKEN AIP(大阪大学 / 丰田公司 lattice 实验室 / RIKEN AIP);Machine Learning Research Group, University of Oxford(机器学习研究组,牛津大学)
Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
University of Tübingen(图宾根大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Oxford(牛津大学);Pennsylvania State University(宾夕法尼亚州立大学)
New Equivalences Between Interpolation and SVMs: Kernels and Structured Features
School of Electrical and Computer Engineering, Georgia Institute of Technology(佐治亚理工学院电子与计算机工程系);Institute of Mathematics, EPFL(瑞士联邦理工学院数学学院);School of Industrial & Systems Engineering, Georgia Institute of Technology(佐治亚理工学院工业与系统工程系)
Equivalence of distance-based and RKHS-based statistics in hypothesis testing
University College London(伦敦大学学院);University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Institute of Statistical Mathematics(统计数学研究所)