In this paper we propose and study a class of simple, nonparametric, yet interpretable measures of conditional dependence between two random variables $Y$ and $Z$ given a third variable $X$, all taking values in general topological spaces. The population version of any of these measures captures the strength of conditional dependence and it is 0 if and only if $Y$ and $Z$ are conditionally independent given $X$, and 1 if and only if $Y$ is a measurable function of $Z$ and $X$. Thus, our measure -- which we call kernel partial correlation (KPC) coefficient -- can be thought of as a nonparametric generalization of the partial correlation coefficient that possesses the above properties when $(X,Y,Z)$ is jointly normal. We describe two consistent methods of estimating KPC. Our first method utilizes the general framework of geometric graphs, including $K$-nearest neighbor graphs and minimum spanning trees. A sub-class of these estimators can be computed in near linear time and converges at a rate that automatically adapts to the intrinsic dimension(s) of the underlying distribution(s). Our second strategy involves direct estimation of conditional mean embeddings using cross-covariance operators in the reproducing kernel Hilbert spaces. Using these empirical measures we develop forward stepwise (high-dimensional) nonlinear variable selection algorithms. We show that our algorithm, using the graph-based estimator, yields a provably consistent model-free variable selection procedure, even in the high-dimensional regime when the number of covariates grows exponentially with the sample size, under suitable sparsity assumptions. Extensive simulation and real-data examples illustrate the superior performance of our methods compared to existing procedures. The recent conditional dependence measure proposed by Azadkia and Chatterjee (2019) can be viewed as a special case of our general framework.
Measuring conditional independence is one of the important tasks in statistical inference and is fundamental in causal discovery, feature selection, dimensionality reduction, Bayesian network learning, and others. In this work, we explore the connection between conditional independence measures induced by distances on a metric space and reproducing kernels associated with a reproducing kernel Hilbert space (RKHS). For certain distance and kernel pairs, we show the distance-based conditional independence measures to be equivalent to that of kernel-based measures. On the other hand, we also show…
A nonparametric family of conditional distributions is introduced, which generalizes conditional exponential families using functional parameters in a suitable RKHS. An algorithm is provided for learning the generalized natural parameter, and consistency of the estimator is established in the well specified case. In experiments, the new method generally outperforms a competing approach with consistency guarantees, and is competitive with a deep conditional density model on datasets that exhibit abrupt transitions and heteroscedasticity.
We present an operator-free, measure-theoretic approach to the conditional mean embedding (CME) as a random variable taking values in a reproducing kernel Hilbert space. While the kernel mean embedding of unconditional distributions has been defined rigorously, the existing operator-based approach of the conditional version depends on stringent assumptions that hinder its analysis. We overcome this limitation via a measure-theoretic treatment of CMEs. We derive a natural regression interpretation to obtain empirical estimates, and provide a thorough theoretical analysis thereof, including univ…
University of Sydney（悉尼大学）；Data61, CSIRO（联邦科学与工业研究组织数据61研究所）；Australian National University（澳大利亚国立大学）
Conditional kernel mean embeddings are nonparametric models that encode conditional expectations in a reproducing kernel Hilbert space. While they provide a flexible and powerful framework for probabilistic inference, their performance is highly dependent on the choice of kernel and regularization hyperparameters. Nevertheless, current hyperparameter tuning methods predominantly rely on expensive cross validation or heuristics that is not optimized for the inference task. For conditional kernel mean embeddings with categorical targets and arbitrary inputs, we propose a hyperparameter learning …
Zalando Research（Zalando研究部）；Zalando SE（Zalando股份公司）；Freie Universität Berlin（柏林自由大学）；MPI for Intelligent Systems（智能系统马克斯·普朗克研究所）
We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean embeddings by drawing connections to estimation of Radon-Nikodym derivatives in the reproducing kernel Hilbert space (RKHS). We prove finite sample bounds for the estimation error in a standard density…
Brown University（布朗大学）；National ICT Australia and ANU（澳大利亚国家信息通信技术中心与澳大利亚国立大学）
In this paper we de ne conditional random elds in reproducing kernel Hilbert spaces and show connections to Gaussian Process classi cation. More speci cally, we prove decomposition results for undirected graphical models and we give constructions for kernels. Finally we present e cient means of solving the optimization problem using reduced rank decompositions and we show how stationarity can be exploited e ciently in the optimization process.
We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as the solution of a global cost minimization procedure over a fixed model nor is it a linear estimator. Instead, approximate solutions are constructed by projections onto a nested set of data-dependent subspaces. To prove consistency, we exploit the known fact that Partial Least S…
Conditional mean embeddings (CMEs) have proven themselves to be a powerful tool in many machine learning applications. They allow the efficient conditioning of probability distributions within the corresponding reproducing kernel Hilbert spaces (RKHSs) by providing a linear-algebraic relation for the kernel mean embeddings of the respective joint and conditional probability distributions. Both centred and uncentred covariance operators have been used to define CMEs in the existing literature. In this paper, we develop a mathematically rigorous theory for both variants, discuss the merits and p…
We propose a new method to test conditional independence of two real random variables $Y$ and $Z$ conditionally on an arbitrary third random variable $X$. %with $F_{.|.}$ representing conditional distribution functions, The partial copula is introduced, defined as the joint distribution of $U=F_{Y|X}(Y|X)$ and $V=F_{Z|X}(Z|X)$. We call this transformation of $(Y,Z)$ into $(U,V)$ the partial copula transform. It is easy to show that if $Y$ and $Z$ are continuous for any given value of $X$, then $Y\ind Z|X$ implies $U\ind V$. Conditional independence can then be tested by (i) applying the partia…
We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate $X$ from the response $Y$, given the projection of $X$ on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316--342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an $M$-estimator for the central subspac…
Under covariate shift, training (source) data and testing (target) data differ in input space distribution, but share the same conditional label distribution. This poses a challenging machine learning task. Robust Bias-Aware (RBA) prediction provides the conditional label distribution that is robust to the worstcase logarithmic loss for the target distribution while matching feature expectation constraints from the source distribution. However, employing RBA with insufficient feature constraints may result in high certainty predictions for much of the source data, while leaving too much uncert…
We study nonparametric estimation for the partially conditional average treatment effect, defined as the treatment effect function over an interested subset of confounders. We propose a hybrid kernel weighting estimator where the weights aim to control the balancing error of any function of the confounders from a reproducing kernel Hilbert space after kernel smoothing over the subset of interested variables. In addition, we present an augmented version of our estimator which can incorporate estimations of outcome mean functions. Based on the representer theorem, gradient-based algorithms can b…
Constraint-based causal discovery (CCD) algorithms require fast and accurate conditional independence (CI) testing. The Kernel Conditional Independence Test (KCIT) is currently one of the most popular CI tests in the non-parametric setting, but many investigators cannot use KCIT with large datasets because the test scales cubicly with sample size. We therefore devise two relaxations called the Randomized Conditional Independence Test (RCIT) and the Randomized conditional Correlation Test (RCoT) which both approximate KCIT by utilizing random Fourier features. In practice, both of the proposed …
This paper introduces new efficient algorithms for two problems: sampling conditional on vertex degrees in unweighted graphs, and sampling conditional on vertex strengths in weighted graphs. The algorithms can sample conditional on the presence or absence of an arbitrary number of edges. The resulting conditional distributions provide the basis for exact tests. Existing samplers based on MCMC or sequential importance sampling are generally not scalable; their efficiency degrades in sparse graphs. MCMC methods usually require explicit computation of a Markov basis to navigate the complex state …
It is well known that the dependence structure for jointly Gaussian variables can be fully captured using correlations, and that the conditional dependence structure in the same way can be described using partial correlations. The partial orrelation does not, however, characterize conditional dependence in many non-Gaussian populations. This paper introduces the local Gaussian partial correlation (LGPC), a new measure of conditional dependence. It is a local version of the partial correlation coefficient that characterizes conditional dependence in a large class of populations. It has some use…
We study nonparametric estimators of conditional Kendall's tau, a measure of concordance between two random variables given some covariates. We prove non-asymptotic bounds with explicit constants, that hold with high probabilities. We provide "direct proofs" of the consistency and the asymptotic law of conditional Kendall's tau. A simulation study evaluates the numerical performance of such nonparametric estimators.
Renewable energy is essential for energy security and global warming mitigation. However, power generation from renewable energy sources is uncertain due to volatile weather conditions and complex equipment operations. To improve equipment's operation efficiency, it is important to understand and characterize the uncertainty in renewable power generation. In this paper, we proposed a conditional kernel density estimation method to model the distribution of equipment's power output given any weather conditions. It explicitly accounts for the temporal dependence in the data stream and uses an it…
We propose a method for feature selection that employs kernel-based measures of independence to find a subset of covariates that is maximally predictive of the response. Building on past work in kernel dimension reduction, we show how to perform feature selection via a constrained optimization problem involving the trace of the conditional covariance operator. We prove various consistency results for this procedure, and also demonstrate that our method compares favorably with other state-of-the-art algorithms on a variety of synthetic and real data sets.
This paper, investigates the conditional quantile estimation of a scalar random response and a functional random covariate (i.e. valued in some infinite-dimensional space) whenever {\it functional stationary ergodic data with random censorship} are considered. We introduce a kernel type estimator of the conditional quantile function. We establish the strong consistency with rate of this estimator as well as the asymptotic normality which induces a confidence interval that is usable in practice since it does not depend on any unknown quantity. An application to electricity peak demand interval …
We propose a kernel-based partial permutation test for checking the equality of functional relationship between response and covariates among different groups. The main idea, which is intuitive and easy to implement, is to keep the projections of the response vector $\boldsymbol{Y}$ on leading principle components of a kernel matrix fixed and permute $\boldsymbol{Y}$'s projections on the remaining principle components. The proposed test allows for different choices of kernels, corresponding to different classes of functions under the null hypothesis. First, using linear or polynomial kernels, …
Sequential Kernel-based Conditional Independence Testing via Adaptive Betting
Department of Computer Science, University of British Columbia, Vancouver, Canada（不列颠哥伦比亚大学计算机科学系，温哥华，加拿大）；Alberta Machine Intelligence Institute, Edmonton, Canada（阿尔伯塔机器智能研究所，爱德蒙顿，加拿大）
Conditional independence of treatment assignment from potential outcomes is a commonly used but nonrefutable assumption. We derive identified sets for various treatment effect parameters under nonparametric deviations from this conditional independence assumption. These deviations are defined via a conditional treatment assignment probability, which makes it straightforward to interpret. Our results can be used to assess the robustness of empirical conclusions obtained under the baseline conditional independence assumption.
This paper introduces the \textit{weighted partial copula} function for testing conditional independence. The proposed test procedure results from these two ingredients: (i) the test statistic is an explicit Cramer-von Mises transformation of the \textit{weighted partial copula}, (ii) the regions of rejection are computed using a bootstrap procedure which mimics conditional independence by generating samples from the product measure of the estimated conditional marginals. Under conditional independence, the weak convergence of the \textit{weighted partial copula proces}s is established when th…
A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)
In this paper, nonparametric estimation of the conditional Weibull-tail coefficient when the variable of interest is right random censored is addressed. A Weissman-type estimator of conditional extreme quantile is also proposed. In addition, a simulation study is conducted to assess the finite-sample behavior of the proposed estimators and a comparison with alternative strategies is provided. Finally, the practical applicability of the methodology is presented using a real datasets of men suffering from a larynx cancer.
We extend path analysis by giving sufficient conditions for computing the partial covariance of two random variables from their covariance. This is specifically done by correcting the covariance with the product of some partial variance ratios. As a result, the partial covariance retains the covariance's salient feature of factorizing over the edges in the paths between the two variables of interest.
We present a new non-parametric estimator of the conditional density of the kernel type. It is based on an efficient transformation of the data by quantile transform. By use of the copula representation, it turns out to have a remarkable product form. We study its asymptotic properties and compare its bias and variance to competitors based on nonparametric regression.
