This article studies global testing of the slope function in functional linear regression model in the framework of reproducing kernel Hilbert space. We propose a new testing statistic based on smoothness regularization estimators. The asymptotic distribution of the testing statistic is established under null hypothesis. It is shown that the null asymptotic distribution is determined jointly by the reproducing kernel and the covariance function. Our theoretical analysis shows that the proposed testing is consistent over a class of smooth local alternatives. Despite the generality of the method of regularization, we show the procedure is easily implementable. Numerical examples are provided to demonstrate the empirical advantages over the competing methods.
A Hilbert space embedding for probability measures has recently been proposed, wherein any probability measure is represented as a mean element in a reproducing kernel Hilbert space (RKHS). Such an embedding has found applications in homogeneity testing, independence testing, dimensionality reduction, etc., with the requirement that the reproducing kernel is characteristic, i.e., the embedding is injective. In this paper, we generalize this embedding to finite signed Borel measures, wherein any finite signed Borel measure is represented as a mean element in an RKHS. We show that the proposed e…
An extension of reproducing kernel Hilbert space (RKHS) theory provides a new framework for modeling functional regression models with functional responses. The approach only presumes a general nonlinear regression structure as opposed to previously studied linear regression models. Generalized cross-validation (GCV) is proposed for automatic smoothing parameter estimation. The new RKHS estimate is applied to both simulated and real data as illustrations.
We develop methodology for testing hypotheses regarding the slope function in functional linear regression for time series via a reproducing kernel Hilbert space approach. In contrast to most of the literature, which considers tests for the exact nullity of the slope function, we are interested in the null hypothesis that the slope function vanishes only approximately, where deviations are measured with respect to the $L^2$-norm. An asymptotically pivotal test is proposed, which does not require the estimation of nuisance parameters and long-run covariances. The key technical tools to prove th…
In this paper we derive the asymptotic distributions of two distinct regularized estimators for functional canonical correlation as well as their associated eigenvalues, eigenvectors and projection operators. The methods we developed utilize regularized estimators which approach the functional operators based in reproducing kernel Hilbert spaces (RKHS) as the regularization parameter approaches zero. In addition to providing some justification for the RKHS methods, we explore the asymptotics of regularized operators associated with both Tikhinov and truncated singular value decomposition (TSVD…
The main ideas behind the classical multivariate logistic regression model make sense when translated to the functional setting, where the explanatory variable $X$ is a function and the response $Y$ is binary. However, some important technical issues appear (or are aggravated with respect to those of the multivariate case) due to the functional nature of the explanatory variable. First, the mere definition of the model can be questioned: while most approaches so far proposed rely on the $L_2$-based model, we suggest an alternative (in some sense, more general) approach, based on the theory of …
We propose a nonlinear function-on-function regression model where both the covariate and the response are random functions. The nonlinear regression is carried out in two steps: we first construct Hilbert spaces to accommodate the functional covariate and the functional response, and then build a second-layer Hilbert space for the covariate to capture nonlinearity. The second-layer space is assumed to be a reproducing kernel Hilbert space, which is generated by a positive definite kernel determined by the inner product of the first-layer Hilbert space for $X$--this structure is known as the n…
In this work, we develop and study an empirical projection operator scheme for solving nonparametric regression problems. This scheme is based on an approximate projection of the regression function over a suitable reproducing kernel Hilbert space (RKHS). The RKHS considered in this paper are generated by the Mercer kernels given by the Legendre Christoffel-Darboux and convolution Sinc kernels. We provide error and convergence analysis of the proposed scheme under the assumption that the regression function belongs to some suitable functional spaces. We also consider the popular RKHS regulariz…
Optimal experimental design seeks to determine the most informative allocation of experiments to infer an unknown statistical quantity. In this work, we investigate the optimal design of experiments for {\em estimation of linear functionals in reproducing kernel Hilbert spaces (RKHSs)}. This problem has been extensively studied in the linear regression setting under an estimability condition, which allows estimating parameters without bias. We generalize this framework to RKHSs, and allow for the linear functional to be only approximately inferred, i.e., with a fixed bias. This scenario captur…
RKHS-based Latent Position Random Graph Correlation
Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院）；Department of Statistics, The Chinese University of Hong Kong（香港中文大学统计系）
This paper presents new results on Functional Analysis of Variance for fixed effect models with correlated Hilbert-valued Gaussian error components. The geometry of the Reproducing Kernel Hilbert Space (RKHS) of the error term is considered in the computation of the total sum of squares, the residual sum of squares, and the sum of squares due to the regression. Under suitable linear transformation of the correlated functional data, the distributional characteristics of these statistics, their moment generating and characteristic functions, are derived. Fixed effect linear hypothesis testing is…
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Pennsylvania State University（宾夕法尼亚州立大学）
We study the fair regression problem under the notion of Mean Parity (MP) fairness, which requires the conditional mean of the learned function output to be constant with respect to the sensitive attributes. We address this problem by leveraging reproducing kernel Hilbert space (RKHS) to construct the functional space whose members are guaranteed to satisfy the fairness constraints. The proposed functional space suggests a closed-form solution for the fair regression problem that is naturally compatible with multiple sensitive attributes. Furthermore, by formulating the fairness-accuracy trade…
A Hilbert space embedding for probability measures has recently been proposed, with applications including dimensionality reduction, homogeneity testing, and independence testing. This embedding represents any probability measure as a mean element in a reproducing kernel Hilbert space (RKHS). A pseudometric on the space of probability measures can be defined as the distance between distribution embeddings: we denote this as $γ_k$, indexed by the kernel function $k$ that defines the inner product in the RKHS. We present three theoretical properties of $γ_k$. First, we consider the question of d…
This paper focuses on the application of Discriminant Analysis to a set of geometrical objects (bodies) characterized by currents. A current is a relevant mathematical object to model geometrical data, like hypersurfaces, through integration of vector fields along them. As a consequence of the choice of a vector-valued Reproducing Kernel Hilbert Space (RKHS) as a test space to integrate hypersurfaces, it is possible to consider that hypersurfaces are embedded in this Hilbert space. This embedding enables us to consider classification algorithms of geometrical objects. A method to apply Functio…
Given a reproducing kernel Hilbert space H of real-valued functions and a suitable measure mu over the source space D (subset of R), we decompose H as the sum of a subspace of centered functions for mu and its orthogonal in H. This decomposition leads to a special case of ANOVA kernels, for which the functional ANOVA representation of the best predictor can be elegantly derived, either in an interpolation or regularization framework. The proposed kernels appear to be particularly convenient for analyzing the e ffect of each (group of) variable(s) and computing sensitivity indices without recur…
The Hájek-Feldman dichotomy establishes that two Gaussian measures are either mutually absolutely continuous with respect to each other (and hence there is a Radon-Nikodym density for each measure with respect to the other one) or mutually singular. Unlike the case of finite dimensional Gaussian measures, there are non-trivial examples of both situations when dealing with Gaussian stochastic processes. This paper provides: (a) Explicit expressions for the optimal (Bayes) rule and the minimal classification error probability in several relevant problems of supervised binary classification of mu…
A general formulation of the linear model with functional (random) explanatory variable $X = X(t), t \in T$ , and scalar response Y is proposed. It includes the standard functional linear model, based on the inner product in the space $L^2[0,1]$, as a particular case. It also includes all models in which Y is assumed to be (up to an additive noise) a linear combination of a finite or countable collections of marginal variables X(t_j), with $t_j\in T$ or a linear combination of a finite number of linear projections of X. This general formulation can be interpreted in terms of the RKHS space gen…
University of Oxford（牛津大学）；Microsoft Research Cambridge（微软剑桥研究院）
Feature attribution for kernel methods is often heuristic and not individualised for each prediction. To address this, we turn to the concept of Shapley values~(SV), a coalition game theoretical framework that has previously been applied to different machine learning model interpretation tasks, such as linear models, tree ensembles and deep networks. By analysing SVs from a functional perspective, we propose \textsc{RKHS-SHAP}, an attribution method for kernel machines that can efficiently compute both \emph{Interventional} and \emph{Observational Shapley values} using kernel mean embeddings o…
McGill University, Department of Mathematics and Statistics（麦吉尔大学数学与统计学系）；Université Marie et Louis Pasteur, CNRS, LmB (UMR 6623)（玛丽·路易斯·帕斯特大学、法国国家科学研究中心、LmB（UMR 6623））
This paper develops a frequentist solution to the functional calibration problem, where the value of a calibration parameter in a computer model is allowed to vary with the value of control variables in the physical system. The need of functional calibration is motivated by engineering applications where using a constant calibration parameter results in a significant mismatch between outputs from the computer model and the physical experiment. Reproducing kernel Hilbert spaces (RKHS) are used to model the optimal calibration function, defined as the functional relationship between the calibrat…
Non-asymptotic two-sample kernel testing with the spectrally truncated normalized MMD
Nantes Université（南特大学）；Laboratoire de Mathématiques Jean Leray, LMJL（让·勒雷数学实验室）；Ecole Normale Supérieure de Lyon（里昂高等师范学院）；Université Claude Bernard Lyon 1（里昂第一大学）；Laboratoire de Biologie et Modélisation de la Cellule（细胞生物学与建模实验室）；CEREMADE, CNRS, Université Paris-Dauphine（巴黎道芬大学数学与计算机科学研究与应用中心）；Université PSL（巴黎文理研究大学）；Université Paris-Saclay（巴黎萨克雷大学）；Inria, LMO（法国国家信息与自动化研究所，数学与优化实验室）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Google Research（谷歌研究院）；University of Freiburg（弗赖堡大学）
We propose kernel distributionally robust optimization (Kernel DRO) using insights from the robust optimization theory and functional analysis. Our method uses reproducing kernel Hilbert spaces (RKHS) to construct a wide range of convex ambiguity sets, which can be generalized to sets based on integral probability metrics and finite-order moment bounds. This perspective unifies multiple existing robust and stochastic optimization methods. We prove a theorem that generalizes the classical duality in the mathematical problem of moments. Enabled by this theorem, we reformulate the maximization wi…
Mahalanobis distance is a classical tool in multivariate analysis. We suggest here an extension of this concept to the case of functional data. More precisely, the proposed definition concerns those statistical problems where the sample data are real functions defined on a compact interval of the real line. The obvious difficulty for such a functional extension is the non-invertibility of the covariance operator in infinite-dimensional cases. Unlike other recent proposals, our definition is suggested and motivated in terms of the Reproducing Kernel Hilbert Space (RKHS) associated with the stoc…
The discussion focuses on metric covariance, a new association measure between paired random objects in a metric space, developed by Dubey and Müller, and on its relationship with other similar concepts which have previously appeared in the literature, including distance covariance by Székely et al, as well as its generalisations which rely on the formalism of reproducing kernel Hilbert spaces (RKHS).
LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎综合理工学院，巴黎电信学院，LTCI实验室）
Operator-Valued Kernels (OVKs) and associated vector-valued Reproducing Kernel Hilbert Spaces provide an elegant way to extend scalar kernel methods when the output space is a Hilbert space. Although primarily used in finite dimension for problems like multi-task regression, the ability of this framework to deal with infinite dimensional output spaces unlocks many more applications, such as functional regression, structured output prediction, and structured data representation. However, these sophisticated schemes crucially rely on the kernel trick in the output space, so that most of previous…
Considering two independent Poisson processes, we address the question of testing equality of their respective intensities. We first propose single tests whose test statistics are U-statistics based on general kernel functions. The corresponding critical values are constructed from a non-asymptotic wild bootstrap approach, leading to level αtests. Various choices for the kernel functions are possible, including projection, approximation or reproducing kernels. In this last case, we obtain a parametric rate of testing for a weak metric defined in the RKHS associated with the considered reproduc…
In this article, we consider convergence rates in functional linear regression with functional responses, where the linear coefficient lies in a reproducing kernel Hilbert space (RKHS). Without assuming that the reproducing kernel and the covariate covariance kernel are aligned, or assuming polynomial rate of decay of the eigenvalues of the covariance kernel, convergence rates in prediction risk are established. The corresponding lower bound in rates is derived by reducing to the scalar response case. Simulation studies and two benchmark datasets are used to illustrate that the proposed approa…
NTT Network Service Systems Laboratories, NTT Corporation（NTT网络服务系统实验室，NTT公司）；Graduate School of Science and Technology, Keio University（庆应义塾大学科学与技术研究生院）；Center for Data Science, Ehime University（爱媛大学数据科学中心）；Center for Advanced Intelligence Project, RIKEN（理化学研究所高级智能项目中心）；Faculty of Science and Technology, Keio University（庆应义塾大学科学与技术学部）
Kernel methods have been among the most popular techniques in machine learning, where learning tasks are solved using the property of reproducing kernel Hilbert space (RKHS). In this paper, we propose a novel data analysis framework with reproducing kernel Hilbert $C^*$-module (RKHM) and kernel mean embedding (KME) in RKHM. Since RKHM contains richer information than RKHS or vector-valued RKHS (vvRKHS), analysis with RKHM enables us to capture and extract structural properties in such as functional data. We show a branch of theories for RKHM to apply to data analysis, including the representer…
A hierarchical Vovk-Azoury-Warmuth forecaster with discounting for online regression in RKHS
Institute of Mathematics, Mechanics and Computer Sciences of the Southern Federal University（南方联邦大学数学、力学与计算机科学学院）；Regional Scientific and Educational Mathematical Center of the Southern Federal University（南方联邦大学区域科学与教育数学中心）
We consider learning in decentralized heterogeneous networks: agents seek to minimize a convex functional that aggregates data across the network, while only having access to their local data streams. We focus on the case where agents seek to estimate a regression \emph{function} that belongs to a reproducing kernel Hilbert space (RKHS). To incentivize coordination while respecting network heterogeneity, we impose nonlinear proximity constraints. To solve the constrained stochastic program, we propose applying a functional variant of stochastic primal-dual (Arrow-Hurwicz) method which yields a…
Mahidol University（玛希隆大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Institute of Statistical Mathematics（统计数学研究所）；Pennsylvania State University（宾夕法尼亚州立大学）
A Hilbert space embedding of a distribution---in short, a kernel mean embedding---has recently emerged as a powerful tool for machine learning and inference. The basic idea behind this framework is to map distributions into a reproducing kernel Hilbert space (RKHS) in which the whole arsenal of kernel methods can be extended to probability measures. It can be viewed as a generalization of the original "feature map" common to support vector machines (SVMs) and other kernel methods. While initially closely associated with the latter, it has meanwhile found application in fields ranging from kern…
Under the reproducing kernel Hilbert spaces (RKHS), we consider the penalized least-squares of the partially functional linear models (PFLM), whose predictor contains both functional and traditional multivariate parts, and the multivariate part allows a divergent number of parameters. From the non-asymptotic point of view, we focus on the rate-optimal upper and lower bounds of the prediction error. An exact upper bound for the excess prediction risk is shown in a non-asymptotic form under a more general assumption known as the effective dimension to the model, by which we also show the predict…
University College London（伦敦大学学院）；University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Institute of Statistical Mathematics（统计数学研究所）
We provide a unifying framework linking two classes of statistics used in two-sample and independence testing: on the one hand, the energy distances and distance covariances from the statistics literature; on the other, maximum mean discrepancies (MMD), that is, distances between embeddings of distributions to reproducing kernel Hilbert spaces (RKHS), as established in machine learning. In the case where the energy distance is computed with a semimetric of negative type, a positive definite kernel, termed distance kernel, may be defined such that the MMD corresponds exactly to the energy dista…
Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributions in order to implement the kernel Bayes rule and build sequential data models. It was recently shown that transfer operators such as the Perron-Frobenius or Koopman operator can also be approximated in a similar fashion using covariance and cross-covariance operators and that …
A nonparametric family of conditional distributions is introduced, which generalizes conditional exponential families using functional parameters in a suitable RKHS. An algorithm is provided for learning the generalized natural parameter, and consistency of the estimator is established in the well specified case. In experiments, the new method generally outperforms a competing approach with consistency guarantees, and is competitive with a deep conditional density model on datasets that exhibit abrupt transitions and heteroscedasticity.
This paper deals with the problem of nonparametric independence testing, a fundamental decision-theoretic problem that asks if two arbitrary (possibly multivariate) random variables $X,Y$ are independent or not, a question that comes up in many fields like causality and neuroscience. While quantities like correlation of $X,Y$ only test for (univariate) linear independence, natural alternatives like mutual information of $X,Y$ are hard to estimate due to a serious curse of dimensionality. A recent approach, avoiding both issues, estimates norms of an \textit{operator} in Reproducing Kernel Hilb…
We study the estimation and prediction of functional autoregressive~(FAR) processes, a statistical tool for modeling functional time series data. Due to the infinite-dimensional nature of FAR processes, the existing literature addresses its inference via dimension reduction and theoretical results therein require the (unrealistic) assumption of fully observed functional time series. We propose an alternative inference framework based on Reproducing Kernel Hilbert Spaces~(RKHS). Specifically, a nuclear norm regularization method is proposed for estimating the transition operators of the FAR pro…
