We propose a new family of specification tests called kernel conditional moment (KCM) tests. Our tests are built on a novel representation of conditional moment restrictions in a reproducing kernel Hilbert space (RKHS) called conditional moment embedding (CMME). After transforming the conditional moment restrictions into a continuum of unconditional counterparts, the test statistic is defined as the maximum moment restriction (MMR) within the unit ball of the RKHS. We show that the MMR not only fully characterizes the original conditional moment restrictions, leading to consistency in both hypothesis testing and parameter estimation, but also has an analytic expression that is easy to compute as well as closed-form asymptotic distributions. Our empirical studies show that the KCM test has a promising finite-sample performance compared to existing tests.
This paper addresses the problem of semiparametric efficiency bounds for conditional moment restriction models with different conditioning variables. We characterize such an efficiency bound, that in general is not explicit, as a limit of explicit efficiency bounds for a decreasing sequence of unconditional (marginal) moment restriction models. An iterative procedure for approximating the efficient score when this is not explicit is provided. Our theoretical results complete and extend existing results in the literature, provide new insight for the theory of semiparametric efficiency bounds li…
We address the estimation of "extreme" conditional quantiles i.e. when their order converges to one as the sample size increases. Conditions on the rate of convergence of their order to one are provided to obtain asymptotically Gaussian distributed kernel estimators. A Weissman-type estimator and kernel estimators of the conditional tail-index are derived, permitting to estimate extreme conditional quantiles of arbitrary order.
We propose an estimator of the kernel-based conditional mean dependence measure obtained from an appropriate modification of a naive estimator based on usual empirical estimators. We then get asymptotic normality of this estimator both under conditional mean independence hypothesis and under the alternative hypothesis. A new test for conditional mean independence of random variables valued into Hilbert spaces is then introduced.
Google Research（谷歌研究院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University College London（伦敦大学学院）
We propose two nonparametric statistical tests of goodness of fit for conditional distributions: given a conditional probability density function $p(y|x)$ and a joint sample, decide whether the sample is drawn from $p(y|x)r_x(x)$ for some density $r_x$. Our tests, formulated with a Stein operator, can be applied to any differentiable conditional density model, and require no knowledge of the normalizing constant. We show that 1) our tests are consistent against any fixed alternative conditional model; 2) the statistics can be estimated easily, requiring no density estimation as an intermediate…
We address the estimation of conditional quantiles when the covariate is functional and when the order of the quantiles converges to one as the sample size increases. In a first time, we investigate to what extent these large conditional quantiles can still be estimated through a functional kernel estimator of the conditional survival function. Sufficient conditions on the rate of convergence of their order to one are provided to obtain asymptotically Gaussian distributed estimators. In a second time, basing on these result, a functional Weissman estimator is derived, permitting to estimate la…
Current meta-learning approaches focus on learning functional representations of relationships between variables, i.e. on estimating conditional expectations in regression. In many applications, however, we are faced with conditional distributions which cannot be meaningfully summarized using expectation only (due to e.g. multimodality). Hence, we consider the problem of conditional density estimation in the meta-learning setting. We introduce a novel technique for meta-learning which combines neural representation and noise-contrastive estimation with the established literature of conditional…
Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples from the other. Stein discrepancies can be combined with kernels to define kernelized Stein discrepancies (KSDs). While kernels make Stein discrepancies tractable, they pose several challenges in high dimensions. We introduce kernelized complete conditional Stein discrepancies (KCC-SDs). Complete conditionals turn a multivariate distribution into multiple univariate distributions…
CREST（法国 Crest 研究中心）；ENSAE（法国国家统计与经济管理学院）；Facebook AI Research（脸书人工智能研究院）；Université Paris-Dauphine（巴黎多芬大学）；Technion（以色列理工学院）
We propose a new conditional dependence measure and a statistical test for conditional independence. The measure is based on the difference between analytic kernel embeddings of two well-suited distributions evaluated at a finite set of locations. We obtain its asymptotic distribution under the null hypothesis of conditional independence and design a consistent statistical test from it. We conduct a series of experiments showing that our new test outperforms state-of-the-art methods both in terms of type-I and type-II errors even in the high dimensional setting.
Markov kernels play a decisive role in probability and mathematical statistics theories, and are an extension of the concepts of sigma-field and statistic. Concepts such as independence, sufficiency, completeness, ancillarity or conditional distribution have been extended previously to Markov kernels. In this paper, the concept of conditional expectation of a Markov kernel given another is introduced, setting its first properties. An application to clinical diagnosis is provided, obtaining {an} optimality property of the predictive values of a diagnosis test. In a statistical framework, this n…
Conditional density estimation is a general framework for solving various problems in machine learning. Among existing methods, non-parametric and/or kernel-based methods are often difficult to use on large datasets, while methods based on neural networks usually make restrictive parametric assumptions on the probability densities. Here, we propose a novel method for estimating the conditional density based on score matching. In contrast to existing methods, we employ scalable neural networks, but do not make explicit parametric assumptions on densities. The key challenge in applying score mat…
SZTAKI: Institute for Computer Science and Control（SZTAKI：计算机科学与控制研究所）；Eötvös Loránd Research Network（厄特沃什·罗兰研究网络）
In this paper we suggest two statistical hypothesis tests for the regression function of binary classification based on conditional kernel mean embeddings. The regression function is a fundamental object in classification as it determines both the Bayes optimal classifier and the misclassification probabilities. A resampling based framework is presented and combined with consistent point estimators of the conditional kernel mean map, in order to construct distribution-free hypothesis tests. These tests are introduced in a flexible manner allowing us to control the exact probability of type I e…
Graduate University of Advanced Studies (SOKENDAI)（高等研究生院（SOKENDAI））；The Institute of Statistical Mathematics（统计数学研究所）；The University of Adelaide（阿德莱德大学）
This paper, investigates the conditional quantile estimation of a scalar random response and a functional random covariate (i.e. valued in some infinite-dimensional space) whenever {\it functional stationary ergodic data with random censorship} are considered. We introduce a kernel type estimator of the conditional quantile function. We establish the strong consistency with rate of this estimator as well as the asymptotic normality which induces a confidence interval that is usable in practice since it does not depend on any unknown quantity. An application to electricity peak demand interval …
This paper studies the estimation of the conditional density f (x, $\times$) of Y i given X i = x, from the observation of an i.i.d. sample (X i , Y i) $\in$ R d , i = 1,. .. , n. We assume that f depends only on r unknown components with typically r d. We provide an adaptive fully-nonparametric strategy based on kernel rules to estimate f. To select the bandwidth of our kernel rule, we propose a new fast iterative algorithm inspired by the Rodeo algorithm (Wasserman and Lafferty (2006)) to detect the sparsity structure of f. More precisely, in the minimax setting, our pointwise estimator, whi…
This paper introduces the kernel mixture network, a new method for nonparametric estimation of conditional probability densities using neural networks. We model arbitrarily complex conditional densities as linear combinations of a family of kernel functions centered at a subset of training points. The weights are determined by the outer layer of a deep neural network, trained by minimizing the negative log likelihood. This generalizes the popular quantized softmax approach, which can be seen as a kernel mixture network with square and non-overlapping kernels. We test the performance of our met…
The most direct way to express arbitrary dependencies in datasets is to estimate the joint distribution and to apply afterwards the argmax-function to obtain the mode of the corresponding conditional distribution. This method is in practice difficult, because it requires a global optimization of a complicated function, the joint distribution by fixed input variables. This article proposes a method for finding global maxima if the joint distribution is modeled by a kernel density estimation. Some experiments show advantages and shortcomings of the resulting regression method in comparison to th…
The paper considers probability distribution, density, conditional distribution and density and conditional moments as well as their kernel estimators in spaces of generalized functions. This approach does not require restrictions on classes of distributions common in nonparametric estimation. Density in usual function spaces is not well-posed; this paper establishes existence and well-posedness of the generalized density function. It also demonstrates root-n convergence of the kernel density estimator in the space of generalized functions. It is shown that the usual kernel estimator of the co…
The classical approach to multivariate extreme value modelling assumes that the joint distribution belongs to a multivariate domain of attraction. This requires each marginal distribution be individually attracted to a univariate extreme value distribution. An apparently more flexible extremal model for multivariate data was proposed by Heffernan and Tawn under which not all the components are required to belong to an extremal domain of attraction but assumes instead the existence of an asymptotic approximation to the conditional distribution of the random vector given one of the components is…
Kernel conditional tests from learning-theoretic bounds
Institute for Data Science in Mechanical Engineering, RWTH Aachen University（机械工程数据科学研究所，亚琛工业大学）；Department of Mathematics, School of Computation, Information and Technology, Technical University of Munich（数学系，计算、信息与技术学院，慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Kernel-Based Evaluation of Conditional Biological Sequence Models
Systems Biology, Harvard Medical School, Boston, USA（系统生物学，哈佛医学院，美国波士顿）；Harvard Medical School, Broad Institute, Boston, USA（哈佛医学院，Broad研究所，美国波士顿）；Department of Statistics, University of Oxford, Oxford, UK（统计学系，牛津大学，英国牛津）；Courant Institute, New York University, New York, USA（Courant研究所，纽约大学，美国纽约）
Nonparametric regression quantiles obtained by inverting a kernel estimator of the conditional distribution of the response are long established in statistics. Attention has been, however, restricted to ordinary quantiles staying away from the tails of the conditional distribution. The purpose of this paper is to extend their asymptotic theory far enough into the tails. We focus on extremal quantile regression estimators of a response variable given a vector of covariates in the general setting, whether the conditional extreme-value index is positive, negative, or zero. Specifically, we elucid…
Renewable energy is essential for energy security and global warming mitigation. However, power generation from renewable energy sources is uncertain due to volatile weather conditions and complex equipment operations. To improve equipment's operation efficiency, it is important to understand and characterize the uncertainty in renewable power generation. In this paper, we proposed a conditional kernel density estimation method to model the distribution of equipment's power output given any weather conditions. It explicitly accounts for the temporal dependence in the data stream and uses an it…
Conditional expectiles are becoming an increasingly important tool in finance as well as in other areas of applications. We analyse a support vector machine type approach for estimating conditional expectiles and establish learning rates that are minimax optimal modulo a logarithmic factor if Gaussian RBF kernels are used and the desired expectile is smooth in a Besov sense. As a special case, our learning rates improve the best known rates for kernel-based least squares regression in this scenario. Key ingredients of our statistical analysis are a general calibration inequality for the asymme…
We find the asymptotic distribution of the multi-dimensional multi-scale and kernel estimators for high-frequency financial data with microstructure. Sampling times are allowed to be asynchronous and endogenous. In the process, we show that the classes of multi-scale and kernel estimators for smoothing noise perturbation are asymptotically equivalent in the sense of having the same asymptotic distribution for corresponding kernel and weight functions. The theory leads to multi-dimensional stable central limit theorems and feasible versions. Hence they allow to draw statistical inference for a …
This paper considers extensions of minimum-disparity estimators to the problem of estimating parameters in a regression model that is conditionally specified; that is where a parametric model describes the distribution of a response $y$ conditional on covariates $x$ but does not specify the distribution of $x$. We define these estimators by estimating a non-parametric conditional density estimates and minimizing a disparity between this estimate and the parametric model averaged over values of $x$. The consistency and asymptotic normality of such estimators is demonstrated for a broad class of…
In this paper, nonparametric estimation of the conditional Weibull-tail coefficient when the variable of interest is right random censored is addressed. A Weissman-type estimator of conditional extreme quantile is also proposed. In addition, a simulation study is conducted to assess the finite-sample behavior of the proposed estimators and a comparison with alternative strategies is provided. Finally, the practical applicability of the methodology is presented using a real datasets of men suffering from a larynx cancer.
The recent advent of smart meters has led to large micro-level datasets. For the first time, the electricity consumption at individual sites is available on a near real-time basis. Efficient management of energy resources, electric utilities, and transmission grids, can be greatly facilitated by harnessing the potential of this data. The aim of this study is to generate probability density estimates for consumption recorded by individual smart meters. Such estimates can assist decision making by helping consumers identify and minimize their excess electricity usage, especially during peak time…
Given additional distributional information in the form of moment restrictions, kernel density and distribution function estimators with implied generalised empirical likelihood probabilities as weights achieve a reduction in variance due to the systematic use of this extra information. The particular interest here is the estimation of densities or distributions of (generalised) residuals in semi-parametric models defined by a finite number of moment restrictions. Such estimates are of great practical interest, being potentially of use for diagnostic purposes, including tests of parametric ass…
Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机过程研究所）；Eidgenössische Technische Hochschule Zürich（苏黎世联邦理工学院）
Measuring conditional independence is one of the important tasks in statistical inference and is fundamental in causal discovery, feature selection, dimensionality reduction, Bayesian network learning, and others. In this work, we explore the connection between conditional independence measures induced by distances on a metric space and reproducing kernels associated with a reproducing kernel Hilbert space (RKHS). For certain distance and kernel pairs, we show the distance-based conditional independence measures to be equivalent to that of kernel-based measures. On the other hand, we also show…
Zalando Research（Zalando研究部）；Zalando SE（Zalando股份公司）；Freie Universität Berlin（柏林自由大学）；MPI for Intelligent Systems（智能系统马克斯·普朗克研究所）
We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean embeddings by drawing connections to estimation of Radon-Nikodym derivatives in the reproducing kernel Hilbert space (RKHS). We prove finite sample bounds for the estimation error in a standard density…
Shape restrictions have played a central role in economics as both testable implications of theory and sufficient conditions for obtaining informative counterfactual predictions. In this paper we provide a general procedure for inference under shape restrictions in identified and partially identified models defined by conditional moment restrictions. Our test statistics and proposed inference methods are based on the minimum of the generalized method of moments (GMM) objective function with and without shape restrictions. Uniformly valid critical values are obtained through a bootstrap procedu…
Two known results on the relationship between conditional and unconditional independence are obtained as a consequence of the main result of this paper, a theorem that uses independence of Markov kernels to obtain a minimal condition which added to conditional independence implies independence. Some counterexamples and representation results are provided to clarify the concepts introduced and the propositions of the statement of the main theorem. Moreover, conditional independence and the mentioned results are extended to the framework of Markov kernels.
In this paper we study some asymptotic properties of the kernel conditional quantile estimator with randomly left-truncated data which exhibit some kind of dependence. We extend the result obtained by Lemdani, Ould-Saïd and Poulin [16] in the iid case. The uniform strong convergence rate of the estimator under strong mixing hypothesis is obtained.
The present contribution derives an explicit expression for (a version of) every uni- and multi-variate conditional distribution (i.e., Markov kernel) of Archimedean copulas and uses this representation to generalize a recently established result, saying that in the class of multivariate Archimedean copulas standard uniform convergence implies weak convergence of almost all univariate Markov kernels, to arbitrary multivariate Markov kernels. Moreover, we prove that an Archimedean copula is singular if, and only if, almost all uni- and multivariate Markov kernels are singular. These results are…
We study nonparametric estimators of conditional Kendall's tau, a measure of concordance between two random variables given some covariates. We prove non-asymptotic bounds with explicit constants, that hold with high probabilities. We provide "direct proofs" of the consistency and the asymptotic law of conditional Kendall's tau. A simulation study evaluates the numerical performance of such nonparametric estimators.
Determinantal point process have recently been used as models in machine learning and this has raised questions regarding the characterizations of conditional independence. In this paper we investigate characterizations of conditional independence. We describe some conditional independencies through the conditions on the kernel of a determinantal point process, and show many can be obtained using the graph induced by a kernel of the $L$-ensemble.
This paper shows that the Conditional Quantile Treatment Effect on the Treated can be identified using a combination of (i) a conditional Distributional Difference in Differences assumption and (ii) an assumption on the conditional dependence between the change in untreated potential outcomes and the initial level of untreated potential outcomes for the treated group. The second assumption recovers the unknown dependence from the observed dependence for the untreated group. We also consider estimation and inference in the case where all of the covariates are discrete. We propose a uniform infe…
We develop Markov categories as a framework for synthetic probability and statistics, following work of Golubtsov as well as Cho and Jacobs. This means that we treat the following concepts in purely abstract categorical terms: conditioning and disintegration; various versions of conditional independence and its standard properties; conditional products; almost surely; sufficient statistics; versions of theorems on sufficient statistics due to Fisher--Neyman, Basu, and Bahadur. Besides the conceptual clarity offered by our categorical setup, its main advantage is that it provides a uniform trea…
A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)
Kernel analog forecasting (KAF), alternatively known as kernel principal component regression, is a kernel method used for nonparametric statistical forecasting of dynamically generated time series data. This paper synthesizes descriptions of kernel methods and Koopman operator theory in order to provide a single consistent account of KAF. The framework presented here illuminates the property of the KAF method that, under measure-preserving and ergodic dynamics, it consistently approximates the conditional expectation of observables that are acted upon by the Koopman operator of the dynamical …
We present a new methodology for sufficient dimension reduction (SDR). Our methodology derives directly from the formulation of SDR in terms of the conditional independence of the covariate $X$ from the response $Y$, given the projection of $X$ on the central subspace [cf. J. Amer. Statist. Assoc. 86 (1991) 316--342 and Regression Graphics (1998) Wiley]. We show that this conditional independence assertion can be characterized in terms of conditional covariance operators on reproducing kernel Hilbert spaces and we show how this characterization leads to an $M$-estimator for the central subspac…
