Estimating the score, i.e., the gradient of log density function, from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models that involve flexible yet intractable densities. Kernel estimators based on Stein's methods or score matching have shown promise, however their theoretical properties and relationships have not been fully-understood. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. It allows us to analyse existing estimators and construct new ones with desirable properties by choosing different hypothesis spaces and regularizers. A unified convergence analysis is provided for such estimators. Finally, we propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
University of Tennessee Knoxville（田纳西大学诺克斯维尔分校）；University of Southern California（南加州大学）
The weighted nearest neighbors (WNN) estimator has been popularly used as a flexible and easy-to-implement nonparametric tool for mean regression estimation. The bagging technique is an elegant way to form WNN estimators with weights automatically generated to the nearest neighbors; we name the resulting estimator as the distributional nearest neighbors (DNN) for easy reference. Yet, there is a lack of distributional results for such estimator, limiting its application to statistical inference. Moreover, when the mean regression function has higher-order smoothness, DNN does not achieve the op…
SISSA（国际高等研究学院）；INFN Sezione di Trieste（意大利国家核物理研究所的里雅斯特分部）；University of Camerino（卡梅里诺大学）
We introduce a new nonparametric density estimator inspired by Markov Chains, and generalizing the well-known Kernel Density Estimator (KDE). Our estimator presents several benefits with respect to the usual ones and can be used straightforwardly as a foundation in all density-based algorithms. We prove the consistency of our estimator and we find it typically outperforms KDE in situations of large sample size and high dimensionality. We also employ our density estimator to build a local outlier detector, showing very promising results when applied to some realistic datasets.
This paper concerns the estimation of the regression function at a given point in nonparametric heteroscedastic models with Gaussian noise or with noise having unknown distribution. In the two cases an asymptotically efficient kernel estimator is constructed for the minimax absolute error risk.
An aggregated method of nonparametric estimators based on time-domain and state-domain estimators is proposed and studied. To attenuate the curse of dimensionality, we propose a factor modeling strategy. We first investigate the asymptotic behavior of nonparametric estimators of the volatility matrix in the time domain and in the state domain. Asymptotic normality is separately established for nonparametric estimators in the time domain and state domain. These two estimators are asymptotically independent. Hence, they can be combined, through a dynamic weighting scheme, to improve the efficien…
We investigate nonparametric estimation of a monotone baseline hazard and a decreasing baseline density within the Cox model. Two estimators of a nondecreasing baseline hazard function are proposed. We derive the nonparametric maximum likelihood estimator and consider a Grenander type estimator, defined as the left-hand slope of the greatest convex minorant of the Breslow estimator. We demonstrate that the two estimators are strong consistent and asymptotically equivalent and derive their common limit distribution at a fixed point. Both estimators of a nonincreasing baseline hazard and their a…
A Bayesian nonparametric estimator to entropy is proposed. The derivation of the new estimator relies on using the Dirichlet process and adapting the well-known frequentist estimators of Vasicek (1976) and Ebrahimi, Pflughoeft and Soofi (1994). Several theoretical properties, such as consistency, of the proposed estimator are obtained. The quality of the proposed estimator has been investigated through several examples, in which it exhibits excellent performance.
We consider nonparametric estimation of $L_2$, Renyi-$α$ and Tsallis-$α$ divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of $n^{-1/2}$ when the densities' smoothness, $s$, are both at least $d/4$ where $d$ is the dimension. We also derive minimax lower bounds for this problem which con…
The construction and theoretical analysis of the most popular universally consistent nonparametric density estimators hinge on one functional property: smoothness. In this paper we investigate the theoretical implications of incorporating a multi-view latent variable model, a type of low-rank model, into nonparametric density estimation. To do this we perform extensive analysis on histogram-style estimators that integrate a multi-view model. Our analysis culminates in showing that there exists a universally consistent histogram-style estimator that converges to any multi-view model with a fini…
For the problem of nonparametric estimation of signal in Gaussian noise we point out the strong asymptotically minimax estimators on maxisets for linear estimators (see \cite{ker93,rio}). It turns out that the order of rates of convergence of Pinsker estimator on this maxisets is worse than the order of rates of convergence for the class of linear estimators considered on this maxisets. We show that balls in Sobolev spaces are maxisets for Pinsker estimators.
New local linear estimators are proposed for a wide class of nonparametric regression models. The estimators are uniformly consistent regardless of satisfying traditional conditions of depen\-dence of design elements. The estimators are the solutions of a specially weighted least-squares method. The design can be fixed or random and does not need to meet classical regularity or independence conditions. As an application, several estimators are constructed for the mean of dense functional data. The theoretical results of the study are illustrated by simulations. An example of processing real me…
Sobolev quantities (norms, inner products, and distances) of probability density functions are important in the theory of nonparametric statistics, but have rarely been used in practice, partly due to a lack of practical estimators. They also include, as special cases, $L^2$ quantities which are used in many applications. We propose and analyze a family of estimators for Sobolev quantities of unknown probability density functions. We bound the bias and variance of our estimators over finite samples, finding that they are generally minimax rate-optimal. Our estimators are significantly more com…
We derive asymptotic normality of kernel type deconvolution density estimators. In particular we consider deconvolution problems where the known component of the convolution has a symmetric lambda-stable distribution, 0<lambda<= 2. It turns out that the limit behavior changes if the exponent parameter lambda passes the value one, the case of Cauchy deconvolution.
We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in $d$-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accuracy of every NPMLE. Our results imply, in particular, that every NPMLE achieves near parametric risk (up to logarithmic multiplicative factors) when the true density is a discrete Gaussian mixture without any prior information on the number of mixture components. NPMLEs can natural…
Given a sample from a discretely observed compound Poisson process, we consider estimation of the density of the jump sizes. We propose a kernel type nonparametric density estimator and study its asymptotic properties. An order bound for the bias and an asymptotic expansion of the variance of the estimator are given. Pointwise weak consistency and asymptotic normality are established. The results show that, asymptotically, the estimator behaves very much like an ordinary kernel estimator.
We propose two classes of nonparametric point estimators of $θ=P(X<Y)$ in the case where $(X,Y)$ are paired, possibly dependent, absolutely continuous random variables. The proposed estimators are based on nonparametric estimators of the joint density of $(X,Y)$ and the distribution function of $Z=Y - X$. We explore the use of several density and distribution function estimators and characterise the convergence of the resulting estimators of $θ$. We consider the use of bootstrap methods to obtain confidence intervals. The performance of these estimators is illustrated using simulated and real …
University of Electronic Science and Technology of China（电子科技大学）；Princeton University（普林斯顿大学）
This paper considers the problem of estimation of the Fisher information for location from a random sample of size $n$. First, an estimator proposed by Bhattacharya is revisited and improved convergence rates are derived. Second, a new estimator, termed a clipped estimator, is proposed. Superior upper bounds on the rates of convergence can be shown for the new estimator compared to the Bhattacharya estimator, albeit with different regularity conditions. Third, both of the estimators are evaluated for the practically relevant case of a random variable contaminated by Gaussian noise. Moreover, u…
This paper establishes a necessary and sufficient condition for the asymptotic normality of the nonparametric estimator of sample coverage proposed by Good [Biometrica 40 (1953) 237--264]. This new necessary and sufficient condition extends the validity of the asymptotic normality beyond the previously proven cases.
We provide new asymptotic theory for kernel density estimators, when these are applied to autoregressive processes exhibiting moderate deviations from a unit root. This fills a gap in the existing literature, which has to date considered only nearly integrated and stationary autoregressive processes. These results have applications to nonparametric predictive regression models. In particular, we show that the null rejection probability of a nonparametric t test is controlled uniformly in the degree of persistence of the regressor. This provides a rigorous justification for the validity of the …
In this paper, we consider a piecewise deterministic Markov process (PDMP), with known flow and deterministic transition measure, and unknown jump rate $λ$. To estimate nonparametrically the jump rate, we first construct an adaptive estimator of the stationary density, then we derive a quotient estimator $\hatλ_n$ of $λ$. We provide uniform bounds for the risk of these estimators, and prove that the estimator of the jump rate is nearly minimax (up to a $\ln^2(n)$ factor). Simulations illustrate the behavior of our estimator.
We study the nonparametric Nadaraya-Watson estimator of the drift function for ergodic stochastic processes driven by fractional Brownian motion of Hurst parameter H > 1/2. The estimator is based on the discretely observed stochastic processes. By using the ergodic properties and stochastic integral, we obtain the consistency of the proposed estimator.
A case-control family study is a study where individuals with a disease of interest (case probands) and individuals without the disease (control probands) are randomly sampled from a well-defined population. Possibly right-censored age at onset and disease status are observed for both probands and their relatives. Correlation among the outcomes within a family is induced by factors such as inherited genetic susceptibility, shared environment, and common behavior patterns. For this setting, we present a nonparametric estimator of the marginal survival function, based on local linear estimation …
In this paper, we introduce a new estimator for the emission densities of a nonparametric hidden Markov model. It is adaptive and minimax with respect to each state's regularity--as opposed to globally minimax estimators, which adapt to the worst regularity among the emission densities. Our method is based on Goldenshluger and Lepski's methodology. It is computationally efficient and only requires a family of preliminary estimators, without any restriction on the type of estimators considered. We present two such estimators that allow to reach minimax rates up to a logarithmic term: a spectral…
We study a class of nonlinear nonparametric inverse problems. Specifically, we propose a nonparametric estimator of the dynamics of a monotonically increasing trajectory defined on a finite time interval. Under suitable regularity conditions, we prove consistency of the proposed estimator and show that in terms of $L^2$-loss, the optimal rate of convergence for the proposed estimator is the same as that for the estimation of the derivative of a trajectory. This is a new contribution to the area of nonlinear nonparametric inverse problems. We conduct a simulation study to examine the finite sam…
In this paper, we consider moderate deviations for Good's coverage estimator. The moderate deviation principle and the self-normalized moderate deviation principle for Good's coverage estimator are established. The results are also applied to the hypothesis testing problem and the confidence interval for the coverage.
A new nonparametric estimator of the local Hurst function of a multifractional Gaussian process based on the increment ratio (IR) statistic is defined. In a general frame, the point-wise and uniform weak and strong consistency and a multidimensional central limit theorem for this estimator are established. Similar results are obtained for a refinement of the generalized quadratic variations (QV) estimator. The example of the multifractional Brownian motion is studied in detail. A simulation study is included showing that the IR-estimator is more accurate than the QV-estimator.
In this paper, we investigate a nonparametric approach to provide a recursive estimator of the transition density of a non-stationary piecewise-deterministic Markov process, from only one observation of the path within a long time. In this framework, we do not observe a Markov chain with transition kernel of interest. Fortunately, one may write the transition density of interest as the ratio of the invariant distributions of two embedded chains of the process. Our method consists in estimating these invariant measures. We state a result of consistency and a central limit theorem under some gen…
Stochastic volatility modelling of financial processes has become increasingly popular. The proposed models usually contain a stationary volatility process. We will motivate and review several nonparametric methods for estimation of the density of the volatility process. Both models based on discretely sampled continuous time processes and discrete time models will be discussed. The key insight for the analysis is a transformation of the volatility density estimation problem to a deconvolution model for which standard methods exist. Three type of nonparametric density estimators are reviewed: …
The discrete kernel method was developed to estimate count data distributions, distinguishing discrete associated kernels based on their asymptotic behaviour. This study investigates the class of discrete asymmetric kernels and their resulting non-consistent estimators, but this theoretical drawback of the estimators is balanced by some interesting features in small/medium samples. The role of modal probability and variance of discrete asymmetric kernels is highlighted to help better understand the performance of these kernels, in particular how the binomial kernel outperforms other asymmetric…
In several different fields, there is interest in analyzing the upper or lower tail quantile of the underlying distribution rather than mean or center quantile. However, the investigation of the tail quantile is difficult because of data sparsity. In this paper, we attempt to develop nonparametric quantile regression for the extremal quantile level. In extremal quantile regression, there are two types of technical conditions of the order of convergence of the quantile level: intermediate order or extreme order. For the intermediate order quantile, the ordinary nonparametric estimator is used. …
Nonparametric Correlation Estimator via Solving Cubic Equations and its Application to Brain Functional Connectivity Analysis
Lijiang Culture and Tourism College（丽江文化旅游学院）；York Neuroimaging Centre, University of York（约克大学神经影像中心）；School of Mathematics, Statistics and Actuarial Science, University of Kent（肯特大学数学、统计与精算科学学院）
In this study, we develop an asymptotic theory of nonparametric regression for a locally stationary functional time series. First, we introduce the notion of a locally stationary functional time series (LSFTS) that takes values in a semi-metric space. Then, we propose a nonparametric model for LSFTS with a regression function that changes smoothly over time. We establish the uniform convergence rates of a class of kernel estimators, the Nadaraya-Watson (NW) estimator of the regression function, and a central limit theorem of the NW estimator.
An extension of the Hawkes model where the productivity is variable is considered. In particular, the case is considered where each point may have its own productivity and a simple analytic formula is derived for the maximum likelihood estimators of these productivities. This estimator is compared with an empirical estimator and ways are explored of stabilizing both estimators by lower truncating, smoothing, and rescaling the estimates. Properties of the estimators are explored in simulations, and the methods are applied to seismological and epidemic datasets to show and quantify substantial v…
A broad class of smooth, possibly data-adaptive nonparametric copula estimators that contains empirical Bernstein copulas introduced by Sancetta and Satchell (and thus the empirical beta copula proposed by Segers, Sibuya and Tsukahara) is studied. Within this class, a subclass of estimators that depend on a scalar parameter determining the amount of marginal smoothing and a functional parameter controlling the shape of the smoothing region is specifically considered. Empirical investigations of the influence of these parameters suggest to focus on two particular data-adaptive smooth copula est…
We propose nonparametric estimators for conditional value-at-risk (CVaR) and conditional expected shortfall (CES) associated with conditional distributions of a series of returns on a financial asset. The return series and the conditioning covariates, which may include lagged returns and other exogenous variables, are assumed to be strong mixing and follow a nonparametric conditional location-scale model. First stage nonparametric estimators for location and scale are combined with a generalized Pareto approximation for distribution tails proposed by Pickands (1975) to give final estimators fo…
We discuss nonparametric tests for parametric specifications of regression quantiles. The test is based on the comparison of parametric and nonparametric fits of these quantiles. The nonparametric fit is a Nadaraya-Watson quantile smoothing estimator. An asymptotic treatment of the test statistic requires the development of new mathematical arguments. An approach that makes only use of plugging in a Bahadur expansion of the nonparametric estimator is not satisfactory. It requires too strong conditions on the dimension and the choice of the bandwidth. Our alternative mathematical approach requi…
In a multiple testing context, we consider a semiparametric mixture model with two components where one component is known and corresponds to the distribution of $p$-values under the null hypothesis and the other component $f$ is nonparametric and stands for the distribution under the alternative hypothesis. Motivated by the issue of local false discovery rate estimation, we focus here on the estimation of the nonparametric unknown component $f$ in the mixture, relying on a preliminary estimator of the unknown proportion $θ$ of true null hypotheses. We propose and study the asymptotic properti…
This paper is concerned with non-parametric estimation of the entropy in ranked set sampling. Theoretical properties of the proposed estimator are studied. The proposed estimator is compared with the rival estimator in simple random sampling. The applications of the proposed estimator to the mutual information estimation as well as estimation of the Kullback-Leibler divergence are provided. Several Monte-Carlo simulation studies are conducted to examine the performance of the estimator. The results are applied to the long-leaf pine (pinus palustris) trees and the body fat percentage data sets …
We study the nonparametric estimators of the infinitesimal coefficients of the second-order jump-diffusion models. Under the mild conditions, we obtain the weak consistency and the asymptotic normalities of the estimators.
In this article, we consider a useful product-limit estimator of distribution function proposed by Huang & Qin(2011) when the observations are subject to length-biased and right-censored data. The estimator retains the simple closed-form expression of the truncation product-limit estimator with some good properties. An almost sure representation for the estimator is obtained which can be used to derive many properties of functional statistics based on this product-limit estimator. The rate for the remainder in the representation is of order $O(n^{-3/4}(\log(n)^{3/4}) $ a.s.
We consider inference procedures, conditional on an observed ancillary statistic, for regression coefficients under a linear regression setup where the unknown error distribution is specified nonparametrically. We establish conditional asymptotic normality of the regression coefficient estimators under regularity conditions, and formally justify the approach of plugging in kernel-type density estimators in conditional inference procedures. Simulation results show that the approach yields accurate conditional coverage probabilities when used for constructing confidence intervals. The plug-in ap…
There is a long-standing debate in the statistical, epidemiological and econometric fields as to whether nonparametric estimation that uses data-adaptive methods, like machine learning algorithms in model fitting, confer any meaningful advantage over simpler, parametric approaches in real-world, finite sample estimation of causal effects. We address the question: when trying to estimate the effect of a treatment on an outcome, across a universe of reasonable data distributions, how much does the choice of nonparametric vs.~parametric estimation matter? Instead of answering this question with s…
We consider nonparametric estimation of a regression curve when the data are observed with multiplicative distortion which depends on an observed confounding variable. We suggest several estimators, ranging from a relatively simple one that relies on restrictive assumptions usually made in the literature, to a sophisticated piecewise approach that involves reconstructing a smooth curve from an estimator of a constant multiple of its absolute value, and which can be applied in much more general scenarios. We show that, although our nonparametric estimators are constructed from predictors of the…
This paper is concerned with nonparametric estimation of the Lévy density of a pure jump Lévy process. The sample path is observed at $n$ discrete instants with fixed sampling interval. We construct a collection of estimators obtained by deconvolution methods and deduced from appropriate estimators of the characteristic function and its first derivative. We obtain a bound for the ${\mathbb L}^2$-risk, under general assumptions on the model. Then we propose a penalty function that allows to build an adaptive estimator. The risk bound for the adaptive estimator is obtained under additional assum…
In this paper, a nonparametric maximum likelihood (ML) estimator for band-limited (BL) probability density functions (pdfs) is proposed. The BLML estimator is consistent and computationally efficient. To compute the BLML estimator, three approximate algorithms are presented: a binary quadratic programming (BQP) algorithm for medium scale problems, a Trivial algorithm for large-scale problems that yields a consistent estimate if the underlying pdf is strictly positive and BL, and a fast implementation of the Trivial algorithm that exploits the band-limited assumption and the Nyquist sampling th…
The subject of robust estimation in time series is widely discussed in literature. One of the approaches is to use GM-estimation. This method incorporates a broad class of nonparametric estimators which under suitable conditions includes estimators robust to outliers in data. For the linear models the sensitivity of GM-estimators to outliers have been studied in the work by Martin and Yohai [5], and influence functionals for this estimator were derived. In this paper we follow this direction and examine the asymptotical properties of the class of M-estimators, which is narrower than the class …
A piecewise-deterministic Markov process is a stochastic process whose behavior is governed by an ordinary differential equation punctuated by random jumps occurring at random times. We focus on the nonparametric estimation problem of the jump rate for such a stochastic model observed within a long time interval under an ergodicity condition. We introduce an uncountable class (indexed by the deterministic flow) of recursive kernel estimates of the jump rate and we establish their strong pointwise consistency as well as their asymptotic normality. We propose to choose among this class the estim…
We derive estimators of the density of the event times of current status data. The estimators are derived for the situations where the distribution of the observation times is known and where this distribution is unknown. The density estimators are constructed from kernel estimators of the density of transformed current status data, which have a distribution similar to uniform deconvolution data. Expansions of the expectation and variance as well as asymptotic normality are derived. A reference density based bandwidth selection method is proposed. A simulated example is presented.
