This study intends to introduce kernel mean embedding of probability measures over infinite-dimensional separable Hilbert spaces induced by functional response statistical models. The embedded function represents the concentration of probability measures in small open neighborhoods, which identifies a pseudo-likelihood and fosters a rich framework for statistical inference. Utilizing Maximum Mean Discrepancy, we devise new tests in functional response models. The performance of new derived tests is evaluated against competitors in three major problems in functional data analysis including function-on-scalar regression, functional one-way ANOVA, and equality of covariance operators.
In this paper, we study the minimax estimation of the Bochner integral $$μ_k(P):=\int_{\mathcal{X}} k(\cdot,x)\,dP(x),$$ also called as the kernel mean embedding, based on random samples drawn i.i.d.~from $P$, where $k:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ is a positive definite kernel. Various estimators (including the empirical estimator), $\hatθ_n$ of $μ_k(P)$ are studied in the literature wherein all of them satisfy $\bigl\| \hatθ_n-μ_k(P)\bigr\|_{\mathcal{H}_k}=O_P(n^{-1/2})$ with $\mathcal{H}_k$ being the reproducing kernel Hilbert space induced by $k$. The main contribution…
We consider the asymptotic normality in $L^2$ of kernel estimators of the long run covariance kernel of stationary functional time series. Our results are established assuming a weakly dependent Bernoulli shift structure for the underlying observations, which contains most stationary functional time series models, under mild conditions. As a corollary, we obtain joint asymptotics for functional principal components computed from empirical long run covariance operators, showing that they have the favorable property of being asymptotically independent.
We address the estimation of "extreme" conditional quantiles i.e. when their order converges to one as the sample size increases. Conditions on the rate of convergence of their order to one are provided to obtain asymptotically Gaussian distributed kernel estimators. A Weissman-type estimator and kernel estimators of the conditional tail-index are derived, permitting to estimate extreme conditional quantiles of arbitrary order.
We address the estimation of conditional quantiles when the covariate is functional and when the order of the quantiles converges to one as the sample size increases. In a first time, we investigate to what extent these large conditional quantiles can still be estimated through a functional kernel estimator of the conditional survival function. Sufficient conditions on the rate of convergence of their order to one are provided to obtain asymptotically Gaussian distributed estimators. In a second time, basing on these result, a functional Weissman estimator is derived, permitting to estimate la…
This work investigates the problem of testing whether $d$ functional random variables are jointly independent using a modified estimator of the $d$-variable Hilbert Schmidt Indepedence Criterion ($d$HSIC) which generalizes HSIC for the case where $d \geq 2$. We then get asymptotic normality of this estimator both under joint independence hypothesis and under the alternative hypothesis. A simulation study shows good performance of the proposed test on finite sample.
The reproducing kernel Hilbert space (RKHS) embedding of distributions offers a general and flexible framework for testing problems in arbitrary domains and has attracted considerable amount of attention in recent years. To gain insights into their operating characteristics, we study here the statistical performance of such approaches within a minimax framework. Focusing on the case of goodness-of-fit tests, our analyses show that a vanilla version of the kernel-embedding based test could be suboptimal, and suggest a simple remedy by moderating the embedding. We prove that the moderated approa…
This paper considers the partially functional linear model (PFLM) where all predictive features consist of a functional covariate and a high dimensional scalar vector. Over an infinite dimensional reproducing kernel Hilbert space, the proposed estimation for PFLM is a least square approach with two mixed regularizations of a function-norm and an $\ell_1$-norm. Our main task in this paper is to establish the minimax rates for PFLM under high dimensional setting, and the optimal minimax rates of estimation is established by using various techniques in empirical process theory for analyzing kerne…
A Dictionary of Closed-Form Kernel Mean Embeddings
University College London（伦敦大学学院）；University of Tübingen（蒂宾根大学）；Lappeenranta–Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学（LUT））；Yahoo Research（雅虎研究院）
In this paper, we establish minimax optimal rates of convergence for prediction in a semi-functional linear model that consists of a functional component and a less smooth nonparametric component. Our results reveal that the smoother functional component can be learned with the minimax rate as if the nonparametric component were known. More specifically, a double-penalized least squares method is adopted to estimate both the functional and nonparametric components within the framework of reproducing kernel Hilbert spaces. By virtue of the representer theorem, an efficient algorithm that requir…
Although recovering an Euclidean distance matrix from noisy observations is a common problem in practice, how well this could be done remains largely unknown. To fill in this void, we study a simple distance matrix estimate based upon the so-called regularized kernel estimate. We show that such an estimate can be characterized as simply applying a constant amount of shrinkage to all observed pairwise distances. This fact allows us to establish risk bounds for the estimate implying that the true distances can be estimated consistently in an average sense as the number of objects increases. In a…
Universität Potsdam（波茨坦大学）；Technische Universität Berlin（柏林工业大学）；École Normale Supérieure de Rennes（雷恩高等师范学院）；Université Paris-Saclay（巴黎-萨克雷大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）
We propose an improved estimator for the multi-task averaging problem, whose goal is the joint estimation of the means of multiple distributions using separate, independent data sets. The naive approach is to take the empirical mean of each data set individually, whereas the proposed method exploits similarities between tasks, without any related information being known in advance. First, for each data set, similar or neighboring means are determined from the data by multiple testing. Then each naive estimator is shrunk towards the local average of its neighbors. We prove theoretically that th…
CREST（法国 Crest 研究中心）；ENSAE（法国国家统计与经济管理学院）；Facebook AI Research（脸书人工智能研究院）；Université Paris-Dauphine（巴黎多芬大学）；Technion（以色列理工学院）
We propose a new conditional dependence measure and a statistical test for conditional independence. The measure is based on the difference between analytic kernel embeddings of two well-suited distributions evaluated at a finite set of locations. We obtain its asymptotic distribution under the null hypothesis of conditional independence and design a consistent statistical test from it. We conduct a series of experiments showing that our new test outperforms state-of-the-art methods both in terms of type-I and type-II errors even in the high dimensional setting.
Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional…
SZTAKI: Institute for Computer Science and Control（SZTAKI：计算机科学与控制研究所）；Eötvös Loránd Research Network（厄特沃什·罗兰研究网络）
In this paper we suggest two statistical hypothesis tests for the regression function of binary classification based on conditional kernel mean embeddings. The regression function is a fundamental object in classification as it determines both the Bayes optimal classifier and the misclassification probabilities. A resampling based framework is presented and combined with consistent point estimators of the conditional kernel mean map, in order to construct distribution-free hypothesis tests. These tests are introduced in a flexible manner allowing us to control the exact probability of type I e…
Estimating the innovation probability density is an important issue in any regression analysis. This paper focuses on functional autoregressive models. A residual-based kernel estimator is proposed for the innovation density. Asymptotic properties of this estimator depend on the average prediction error of the functional autoregressive function. Sufficient conditions are studied to provide strong uniform consistency and asymptotic normality of the kernel density estimator.
This paper introduces an approach for detecting differences in the first-order structures of spatial point patterns. The proposed approach leverages the kernel mean embedding in a novel way by introducing its approximate version tailored to spatial point processes. While the original embedding is infinite-dimensional and implicit, our approximate embedding is finite-dimensional and comes with explicit closed-form formulas. With its help we reduce the pattern comparison problem to the comparison of means in the Euclidean space. Hypothesis testing is based on conducting t-tests on each dimension…
Graduate University of Advanced Studies (SOKENDAI)（高等研究生院（SOKENDAI））；The Institute of Statistical Mathematics（统计数学研究所）；The University of Adelaide（阿德莱德大学）
A Hilbert space embedding for probability measures has recently been proposed, wherein any probability measure is represented as a mean element in a reproducing kernel Hilbert space (RKHS). Such an embedding has found applications in homogeneity testing, independence testing, dimensionality reduction, etc., with the requirement that the reproducing kernel is characteristic, i.e., the embedding is injective. In this paper, we generalize this embedding to finite signed Borel measures, wherein any finite signed Borel measure is represented as a mean element in an RKHS. We show that the proposed e…
An extension of reproducing kernel Hilbert space (RKHS) theory provides a new framework for modeling functional regression models with functional responses. The approach only presumes a general nonlinear regression structure as opposed to previously studied linear regression models. Generalized cross-validation (GCV) is proposed for automatic smoothing parameter estimation. The new RKHS estimate is applied to both simulated and real data as illustrations.
In this article a new family of tests is proposed for the comparison problem of the equality of distribution of two-sample under right censoring scheme. The tests are based on energy distance and kernels mean embedding, are calibrated by permutations and are consistent against all alternatives. The good performance of the new tests in real situations with finite samples is established with a simulation study.
We give a new large deviation inequality for sums of random variables of the form $Z_k = f(X_k,X_t)$ for $k,t\in \mathbb{N}$, $t$ fixed, where the underlying process $X$ is $β$-mixing. The inequality can be used to derive concentration inequalities. We demonstrate its usefulness in the functional kernel regression model of Ferraty et al. (2007) where we study the consistency of dynamic forecasts.
Functional times series have become an integral part of both functional data and time series analysis. This paper deals with the functional autoregressive model of order 1 and the autoregression bootstrap for smooth functions. The regression operator is estimated in the framework developed by Ferraty and Vieu [2004] and Ferraty et al. [2007] which is here extended to the double functional case under an assumption of stationary ergodic data which dates back to Laib and Louani [2010]. The main result of this article is the characterization of the asymptotic consistency of the bootstrapped regres…
Given a class of closed Riemannian manifolds with prescribed geometric conditions, we introduce an embedding of the manifolds into $\ell^2$ based on the heat kernel of the Connection Laplacian associated with the Levi-Civita connection on the tangent bundle. As a result, we can construct a distance in this class which leads to a pre-compactness theorem on the class under consideration.
In this manuscript we present exponential inequalities for spatial lattice processes which take values in a separable Hilbert space and satisfy certain dependence conditions. We consider two types of dependence: spatial data under $α$-mixing conditions and spatial data which satisfies a weak dependence condition introduced by Dedecker and Prieur [2005]. We demonstrate their usefulness in the functional kernel regression model of Ferraty and Vieu [2004] where we study uniform consistency properties of the estimated regression operator on increasing subsets of the underlying function space.
Statistical depth is the act of gauging how representative a point is compared to a reference probability measure. The depth allows introducing rankings and orderings to data living in multivariate, or function spaces. Though widely applied and with much experimental success, little theoretical progress has been made in analysing functional depths. This article highlights how the common $h$-depth and related statistical depths for functional data can be viewed as a kernel mean embedding, a technique used widely in statistical machine learning. This connection facilitates answers to open questi…
A Hilbert space embedding for probability measures has recently been proposed, with applications including dimensionality reduction, homogeneity testing, and independence testing. This embedding represents any probability measure as a mean element in a reproducing kernel Hilbert space (RKHS). A pseudometric on the space of probability measures can be defined as the distance between distribution embeddings: we denote this as $γ_k$, indexed by the kernel function $k$ that defines the inner product in the RKHS. We present three theoretical properties of $γ_k$. First, we consider the question of d…
Mahidol University（玛希隆大学）；Idiap Research Institute（Idiap研究所）；University of Bern（伯尔尼大学）；Centre of Excellence in Mathematics（数学卓越中心）
We focus on kernel methods for set-valued inputs and their application to Bayesian set optimization, notably combinatorial optimization. We investigate two classes of set kernels that both rely on Reproducing Kernel Hilbert Space embeddings, namely the ``Double Sum'' (DS) kernels recently considered in Bayesian set optimization, and a class introduced here called ``Deep Embedding'' (DE) kernels that essentially consists in applying a radial kernel on Hilbert space on top of the canonical distance induced by another kernel such as a DS kernel. We establish in particular that while DS kernels ty…
We propose a nonparametric two-sample test procedure based on Maximum Mean Discrepancy (MMD) for testing the hypothesis that two samples of functions have the same underlying distribution, using kernels defined on function spaces. This construction is motivated by a scaling analysis of the efficiency of MMD-based tests for datasets of increasing dimension. Theoretical properties of kernels on function spaces and their associated MMD are established and employed to ascertain the efficacy of the newly proposed test, as well as to assess the effects of using functional reconstructions based on di…
