In the context of density level set estimation, we study the convergence of general plug-in methods under two main assumptions on the density for a given level $λ$. More precisely, it is assumed that the density (i) is smooth in a neighborhood of $λ$ and (ii) has $γ$-exponent at level $λ$. Condition (i) ensures that the density can be estimated at a standard nonparametric rate and condition (ii) is similar to Tsybakov's margin assumption which is stated for the classification framework. Under these assumptions, we derive optimal rates of convergence for plug-in estimators. Explicit convergence rates are given for plug-in estimators based on kernel density estimators when the underlying measure is the Lebesgue measure. Lower bounds proving optimality of the rates in a minimax sense when the density is Hölder smooth are also provided.
Both Marcinkiewicz-Zygmund strong laws of large numbers (MZ-SLLNs) and ordinary strong laws of large numbers (SLLNs) for plug-in estimators of general statistical functionals are derived. It is used that if a statistical functional is "sufficiently regular", then a (MZ-) SLLN for the estimator of the unknown distribution function yields a (MZ-) SLLN for the corresponding plug-in estimator. It is in particular shown that many L-, V- and risk functionals are "sufficiently regular", and that known results on the strong convergence of the empirical process of α-mixing random variables can be impro…
We consider the problem of estimating the joint distribution function of the event time and a continuous mark variable when the event time is subject to interval censoring case 1 and the continuous mark variable is only observed in case the event occurred before the time of inspection. The nonparametric maximum likelihood estimator in this model is known to be inconsistent. We study two alternative smooth estimators, based on the explicit (inverse) expression of the distribution function of interest in terms of the density of the observable vector. We derive the pointwise asymptotic distributi…
This study investigates the effect of bandwidth selection via a plug-in method on the asymptotic structure of the nonparametric kernel density estimator. We generalise the result of Hall and Kang (2001) and find that the plug-in method has no effect on the asymptotic structure of the estimator up to the order of $O\{(nh_0)^{-1/2}+h_0^L\}=O(n^{-L/(2L+1)})$ for a bandwidth $h_0$ and any kernel order $L$ when the kernel order for pilot estimation $L_p$ is high enough. We also provide the valid Edgeworth expansion up to the order of $O\{(nh_0)^{-1}+h_0^{2L}\}$ and find that, as long as the $L_p$ i…
We consider the problem of estimating smooth integrated functionals of a monotone nonincreasing density $f$ on $[0,\infty)$ using the nonparametric maximum likelihood based plug-in estimator. We find the exact asymptotic distribution of this natural (tuning parameter-free) plug-in estimator, properly normalized. In particular, we show that the simple plug-in estimator is always $\sqrt{n}$-consistent, and is additionally asymptotically normal with zero mean and the semiparametric efficient variance for estimating a subclass of integrated functionals. Compared to the previous results on this top…
Asymptotic properties of three estimators of probability density function of sample maximum $f_{(m)}:=mfF^{m-1}$ are derived, where $m$ is a function of sample size $n$. One of the estimators is the parametrically fitted by the approximating generalized extreme value density function. However, the parametric fitting is misspecified in finite $m$ cases. The misspecification comes from mainly the following two: the difference $m$ and the selected block size $k$, and the poor approximation $f_{(m)}$ to the generalized extreme value density which depends on the magnitude of $m$ and the extreme ind…
Optimal transport maps between two probability distributions $μ$ and $ν$ on $\mathbb{R}^d$ have found extensive applications in both machine learning and statistics. In practice, these maps need to be estimated from data sampled according to $μ$ and $ν$. Plug-in estimators are perhaps most popular in estimating transport maps in the field of computational optimal transport. In this paper, we provide a comprehensive analysis of the rates of convergences for general plug-in estimators defined via barycentric projections. Our main contribution is a new stability estimate for barycentric projectio…
Shannon's entropy is one of the building blocks of information theory and an essential aspect of Machine Learning methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon's entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon's entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all ent…
Asymptotic properties of scatter estimators for elliptical graphical models are studied. Such models impose a given pattern of zeros on the inverse of the shape matrix of an elliptically distributed random vector. In particular, we introduce the class of graphical M-estimators and compare them to plug-in M-estimators. It turns out that, under suitable conditions, both approaches yield the same asymptotic efficiency. Furthermore, the results of this paper apply to both decomposable and non-decomposable graphical models and so generalize the results for decomposable models given by Vogel & Fried…
This paper deals with the problem of estimating the level sets $L(c)= \{F(x) \geq c \}$, with $c \in (0,1)$, of an unknown distribution function $F$ on \mathbb{R}^d_+$. A plug-in approach is followed. That is, given a consistent estimator $F_n$ of $F$, we estimate $L(c)$ by $L_n(c)= \{F_n(x) \geq c \}$. We state consistency results with respect to the Hausdorff distance and the volume of the symmetric difference. These results can be considered as generalizations of results previously obtained, in a bivariate framework, in Di Bernardino et al. (2011). Finally we investigate the effects of scal…
Our investigation concerns the estimation of predictive densities and a study of efficiency as measured by the frequentist risk of such predictive densities with integrated $L_2$ and $L_1$ losses. Our findings relate to a $p-$variate spherically symmetric observable $X \sim p_X(\|x-μ\|^2)$ and the objective of estimating the density of $Y \sim q_Y(\|y-μ\|^2)$ based on $X$. For $L_2$ loss, we describe Bayes estimation, minimum risk equivariant estimation (MRE), and minimax estimation. We focus on the risk performance of the benchmark minimum risk equivariant estimator, plug-in estimators, and p…
Confident prediction is highly relevant in machine learning; for example, in applications such as medical diagnoses, wrong prediction can be fatal. For classification, there already exist procedures that allow to not classify data when the confidence in their prediction is weak. This approach is known as classification with reject option. In the present paper, we provide new methodology for this approach. Predicting a new instance via a confidence set, we ensure an exact control of the probability of classification. Moreover, we show that this methodology is easily implementable and entails at…
In the multiple testing context, a challenging problem is the estimation of the proportion $π_0$ of true-null hypotheses. A large number of estimators of this quantity rely on identifiability assumptions that either appear to be violated on real data, or may be at least relaxed. Under independence, we propose an estimator $\hatπ_0$ based on density estimation using both histograms and cross-validation. Due to the strong connection between the false discovery rate (FDR) and $π_0$, many multiple testing procedures (MTP) designed to control the FDR may be improved by introducing an estimator of $…
The functional delta-method provides a convenient tool for deriving the asymptotic distribution of a plug-in estimator of a statistical functional from the asymptotic distribution of the respective empirical process. Moreover, it provides a tool to derive bootstrap consistency for plug-in estimators from bootstrap consistency of empirical processes. It has recently been shown that the range of applications of the functional delta-method for the asymptotic distribution can be considerably enlarged by employing the notion of quasi-Hadamard differentiability. Here we show in a general setting tha…
In this paper we study the ideal variable bandwidth kernel density estimator introduced by McKay (1993) and Jones, McKay and Hu (1994) and the plug-in practical version of the variable bandwidth kernel estimator with two sequences of bandwidths as in Giné and Sang (2013). Based on the bias and variance analysis of the ideal and true variable bandwidth kernel density estimators, we study the central limit theorems for each of them.
The functional delta-method provides a convenient tool for deriving bootstrap consistency of a sequence of plug-in estimators w.r.t. a given functional from bootstrap consistency of the underlying sequence of estimators. It has recently been shown in Beutner and Zähle (2016) that the range of applications of the functional delta-method for establishing bootstrap consistency in probability of the sequence of plug-in estimators can be considerably enlarged by replacing the usual condition of Hadamard differentiability of the given functional by the weaker condition of quasi-Hadamard differentiab…
Some crucial issues about a recently proposed estimator for the proportion of true null hypotheses ($π_0$) under discrete setup are discussed. An estimator for $π_0$ is introduced under the same setup. The estimator may be seen as a modification of a very popular estimator for $π_0$, originally proposed under the assumption of continuous test statistics. It is shown that adaptive Benjamini-Hochberg procedure remains conservative with the new estimator for $π_0$ being plugged in.
We present a new active learning algorithm based on nonparametric estimators of the regression function. Our investigation provides probabilistic bounds for the rates of convergence of the generalization error achievable by proposed method over a broad class of underlying distributions. We also prove minimax lower bounds which show that the obtained rates are almost tight.
Suppose that we wish to estimate a finite-dimensional summary of one or more function-valued features of an underlying data-generating mechanism under a nonparametric model. One approach to estimation is by plugging in flexible estimates of these features. Unfortunately, in general, such estimators may not be asymptotically efficient, which often makes these estimators difficult to use as a basis for inference. Though there are several existing methods to construct asymptotically efficient plug-in estimators, each such method either can only be derived using knowledge of efficiency theory or i…
Motivated by portfolio allocation and linear discriminant analysis, we consider estimating a functional $\mathbfμ^T \mathbfΣ^{-1} \mathbfμ$ involving both the mean vector $\mathbfμ$ and covariance matrix $\mathbfΣ$. We study the minimax estimation of the functional in the high-dimensional setting where $\mathbfΣ^{-1} \mathbfμ$ is sparse. Akin to past works on functional estimation, we show that the optimal rate for estimating the functional undergoes a phase transition between regular parametric rate and some form of high-dimensional estimation rate. We further show that the optimal rate is at…
Nonparametric density and regression estimators commonly depend on a bandwidth. The asymptotic properties of these estimators have been widely studied when bandwidths are nonstochastic. In practice, however, in order to improve finite sample performance of these estimators, bandwidths are selected by data driven methods, such as cross-validation or plug-in procedures. As a result nonparametric estimators are usually constructed using stochastic bandwidths. In this paper we establish the asymptotic equivalence in probability of local polynomial regression estimators under stochastic and nonstoc…
Let $f$ be a multivariate density and $f\_n$ be a kernel estimate of $f$ drawn from the $n$-sample $X\_1,...,X\_n$ of i.i.d. random variables with density $f$. We compute the asymptotic rate of convergence towards 0 of the volume of the symmetric difference between the $t$-level set $\{f\geq t\}$ and its plug-in estimator $\{f\_n\geq t\}$. As a corollary, we obtain the exact rate of convergence of a plug-in type estimate of the density level set corresponding to a fixed probability for the law induced by $f$.
Support size estimation and the related problem of unseen species estimation have wide applications in ecology and database analysis. Perhaps the most used support size estimator is the Chao estimator. Despite its wide spread use, little is known about its theoretical properties. We analyze the Chao estimator and show that its worst case mean squared error (MSE) is smaller than the MSE of the plug-in estimator by a factor of $\mathcal{O} ((k/n)^4)$, where $k$ is the maximum support size and $n$ is the number of samples. Our main technical contribution is a new method to analyze rational estima…
Partly motivated by entropy-estimation problems in neuroscience, we present a detailed and extensive comparison between some of the most popular and effective entropy estimation methods used in practice: The plug-in method, four different estimators based on the Lempel-Ziv (LZ) family of data compression algorithms, an estimator based on the Context-Tree Weighting (CTW) method, and the renewal entropy estimator. **Methodology. Three new entropy estimators are introduced. For two of the four LZ-based estimators, a bootstrap procedure is described for evaluating their standard error, and a pract…
We present a general framework for studying regularized estimators; such estimators are pervasive in estimation problems wherein "plug-in" type estimators are either ill-defined or ill-behaved. Within this framework, we derive, under primitive conditions, consistency and a generalization of the asymptotic linearity property. We also provide data-driven methods for choosing tuning parameters that, under some conditions, achieve the aforementioned properties. We illustrate the scope of our approach by presenting a wide range of applications.
This paper addresses the problem of an efficient predictive density estimation for the density $q(\|y-θ\|^2)$ of $Y$ based on $X \sim p(\|x-θ\|^2)$ for $y, x, θ\in \mathbb{R}^d$. The chosen criteria are integrated $L_1$ loss given by $L(θ, \hat{q}) \, =\, \int_{\mathbb{R}^d} \big|\hat{q}(y)- q(\|y-θ\|^2) \big| \, dy$, and the associated frequentist risk, for $θ\in Θ$. For absolutely continuous and strictly decreasing $q$, we establish the inevitability of scale expansion improvements $\hat{q}_c(y;X)\,=\, \frac{1}{c^d} q\big(\|y-X\|^2/c^2 \big) $ over the plug-in density $\hat{q}_1$, for a subs…
This article examines density estimation by combining a parametric approach with a nonparametric factor. The plug-in parametric estimator is seen as a crude estimator of the true density and is adjusted by a nonparametric factor. The nonparametric factor is derived by a criterion called local L_2-fitting. A class of estimators that have multiplicative adjustment is provided, including estimators proposed by several authors as special cases, and the asymptotic theories are developed. Theoretical comparison reveals that the estimators in this class are better than, or at least competitive with, …
Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables $\mathbf{X}$ and $\mathbf{Y}$ for two cases: 1) $\mathbf{X}$ and $\mathbf{Y}$ are continuous; 2) $\mathbf{X}$ and $\mathbf{Y}$ …
We study three fundamental statistical-learning problems: distribution estimation, property estimation, and property testing. We establish the profile maximum likelihood (PML) estimator as the first unified sample-optimal approach to a wide range of learning tasks. In particular, for every alphabet size $k$ and desired accuracy $\varepsilon$: $\textbf{Distribution estimation}$ Under $\ell_1$ distance, PML yields optimal $Θ(k/(\varepsilon^2\log k))$ sample complexity for sorted-distribution estimation, and a PML-based estimator empirically outperforms the Good-Turing estimator on the actual dis…
Motivated by questions in lossy data compression and by theoretical considerations, we examine the problem of estimating the rate-distortion function of an unknown (not necessarily discrete-valued) source from empirical data. Our focus is the behavior of the so-called "plug-in" estimator, which is simply the rate-distortion function of the empirical distribution of the observed data. Sufficient conditions are given for its consistency, and examples are provided to demonstrate that in certain cases it fails to converge to the true rate-distortion function. The analysis of its performance is com…
For the univariate current status and, more generally, the interval censoring model, distribution theory has been developed for the maximum likelihood estimator (MLE) and smoothed maximum likelihood estimator (SMLE) of the unknown distribution function, see, e.g., [12], [7], [4], [5], [6], [10], [11] and [8]. For the bivariate current status and interval censoring models distribution theory of this type is still absent and even the rate at which we can expect reasonable estimators to converge is unknown. We define a purely discrete plug-in estimator of the distribution function which locally c…
We study the problem of off-policy evaluation in the multi-armed bandit model with bounded rewards, and develop minimax rate-optimal procedures under three settings. First, when the behavior policy is known, we show that the Switch estimator, a method that alternates between the plug-in and importance sampling estimators, is minimax rate-optimal for all sample sizes. Second, when the behavior policy is unknown, we analyze performance in terms of the competitive ratio, thereby revealing a fundamental gap between the settings of known and unknown behavior policies. When the behavior policy is un…
Independent component analysis (ICA) is linked up with the problem of estimating a non linear functional of a density, for which optimal estimators are well known. The precision of ICA is analyzed from the viewpoint of functional spaces in the wavelet framework. In particular, it is shown that, under Besov smoothness conditions, parametric rate of convergence is achieved by a U-statistic estimator of the wavelet ICA contrast, while the previously introduced plug-in estimator $\hat C^2\_j$, with moderate computational cost, has a rate in $n^{-4s\over 4s+d}$.
This paper highlights a tension between semiparametric efficiency and bootstrap consistency in the context of a canonical semiparametric estimation problem, namely the problem of estimating the average density. It is shown that although simple plug-in estimators suffer from bias problems preventing them from achieving semiparametric efficiency under minimal smoothness conditions, the nonparametric bootstrap automatically corrects for this bias and that, as a result, these seemingly inferior estimators achieve bootstrap consistency under minimal smoothness conditions. In contrast, several "debi…
We propose an approach to the estimation of infinite sets of random vectors. The problem addressed is as follows. Given two infinite sets of random vectors, find a single estimator that estimates vectors from with a controlled associated error. A new theory for the existence and implementation of such an estimator is studied. In particular, we show that the proposed estimator is asymptotically optimal. Moreover, the estimator is determined in terms of pseudo-inverse matrices and, therefore, it always exists.
Confidence sets based on sparse estimators are shown to be large compared to more standard confidence sets, demonstrating that sparsity of an estimator comes at a substantial price in terms of the quality of the estimator. The results are set in a general parametric or semiparametric framework.
We extend balloon and sample-smoothing estimators, two types of variable-bandwidth kernel density estimators, by a shift parameter and derive their asymptotic properties. Our approach facilitates the unified study of a wide range of density estimators which are subsumed under these two general classes of kernel density estimators. We demonstrate our method by deriving the asymptotic bias, variance, and mean (integrated) squared error of density estimators with gamma, log-normal, Birnbaum-Saunders, inverse Gaussian and reciprocal inverse Gaussian kernels. We propose two new density estimators f…
Many standard estimators such as several maximum likelihood estimators or the empirical estimator for any law-invariant convex risk measure are not (qualitatively) robust in the classical sense. However, these estimators may nevertheless satisfy a weak (Krätschmer/Schied/Zähle 2012, 2014) or a local (Zähle 2016) robustness property on relevant sets of distributions. One aim of our paper is to identify sets of local robustness, and to explain the benefit of the knowledge of such sets. For instance, we will be able to demonstrate that many maximum likelihood estimators are robust on their natura…
This paper presents new results on prediction of linear processes in function spaces. The autoregressive Hilbertian process framework of order one (ARH(1) process framework) is adopted. A componentwise estimator of the autocorrelation operator is formulated, from the moment-based estimation of its diagonal coefficients, with respect to the orthogonal eigenvectors of the auto-covariance operator, which are assumed to be known. Mean-square convergence to the theoretical autocorrelation operator, in the space of Hilbert-Schmidt operators, is proved. Consistency then follows in that space. For the…
A scheme for locally adaptive bandwidth selection is proposed which sensitively shrinks the bandwidth of a kernel estimator at lowest density regions such as the support boundary which are unknown to the statistician. In case of a Hölder continuous density, this locally minimax-optimal bandwidth is shown to be smaller than the usual rate, even in case of homogeneous smoothness. Some new type of risk bound with respect to a density-dependent standardized loss of this estimator is established. This bound is fully nonasymptotic and allows to deduce convergence rates at lowest density regions that…
The best-known and most commonly used distribution-property estimation technique uses a plug-in estimator, with empirical frequency replacing the underlying distribution. We present novel linear-time-computable estimators that significantly "amplify" the effective amount of data available. For a large variety of distribution properties including four of the most popular ones and for every underlying distribution, they achieve the accuracy that the empirical-frequency plug-in estimators would attain using a logarithmic-factor more samples. Specifically, for Shannon entropy and a very broad clas…
The problem of estimating the directed information rate between two discrete processes $\{X_n\}$ and $\{Y_n\}$ via the plug-in (or maximum-likelihood) estimator is considered. When the joint process $\{(X_n,Y_n)\}$ is a Markov chain of a given memory length, the plug-in estimator is shown to be asymptotically Gaussian and to converge at the optimal rate $O(1/\sqrt{n})$ under appropriate conditions; this is the first estimator that has been shown to achieve this rate. An important connection is drawn between the problem of estimating the directed information rate and that of performing a hypoth…
We characterize the sets of offspring laws on which the Lotka--Nagaev estimator for the mean of a supercritical Galton--Watson process is qualitatively robust. These are exactly the locally uniformly integrating sets of offspring laws, which may be quite large. If the corresponding global property is assumed instead, we obtain uniform robustness as well. We illustrate both results with a number of concrete examples. As a by-product of the proof we obtain that the Lotka--Nagaev estimator is [locally] uniformly weakly consistent on the respective sets of offspring laws, conditionally on non-exti…
