By a mixture density is meant a density of the form $π_μ(\cdot)=\intπ_θ(\cdot)\timesμ(dθ)$, where $(π_θ)_{θ\inΘ}$ is a family of probability densities and $μ$ is a probability measure on $Θ$. We consider the problem of identifying the unknown part of this model, the mixing distribution $μ$, from a finite sample of independent observations from $π_μ$. Assuming that the mixing distribution has a density function, we wish to estimate this density within appropriate function classes. A general approach is proposed and its scope of application is investigated in the case of discrete distributions. Mixtures of power series distributions are more specifically studied. Standard methods for density estimation, such as kernel estimators, are available in this context, and it has been shown that these methods are rate optimal or almost rate optimal in balls of various smoothness spaces. For instance, these results apply to mixtures of the Poisson distribution parameterized by its mean. Estimators based on orthogonal polynomial sequences have also been proposed and shown to achieve similar rates. The general approach of this paper extends and simplifies such results. For instance, it allows us to prove asymptotic minimax efficiency over certain smoothness classes of the above-mentioned polynomial estimator in the Poisson case. We also study discrete location mixtures, or discrete deconvolution, and mixtures of discrete uniform distributions.
This paper outlines a new nonparametric estimation procedure for unobserved phi-mixing processes. It is assumed that the only information on the stationary hidden states (Xk) is given by the process (Yk), where Yk is a noisy observation of f(Xk). The paper introduces a maximum pseudo-likelihood procedure to estimate the function f and the distribution of the hidden states using blocks of observations of length b. The identifiability of the model is studied in the particular cases b=1 and b=2. The consistency of the estimators of f and of the distribution of the hidden states as the number of o…
Mixture models are regularly used in density estimation applications, but the problem of estimating the mixing distribution remains a challenge. Nonparametric maximum likelihood produce estimates of the mixing distribution that are discrete, and these may be hard to interpret when the true mixing distribution is believed to have a smooth density. In this paper, we investigate an algorithm that produces a sequence of smooth estimates that has been conjectured to converge to the nonparametric maximum likelihood estimator. Here we give a rigorous proof of this conjecture, and propose a new data-d…
Discontinuity in density functions is of economic importance and interest. For instance, in studies on regression discontinuity designs, discontinuity in the density of a running variable suggests violation of the no-manipulation assumption. In this paper we develop estimation and testing procedures on discontinuity in densities with positive support. Our approach is built on splitting the gamma kernel (Chen, 2000) into two parts at a given (dis)continuity point and constructing two truncated kernels. The jump-size magnitude of the density at the point can be estimated nonparametrically by two…
We consider the problem of estimating the mixing density $f$ from $n$ i.i.d. observations distributed according to a mixture density with unknown mixing distribution. In contrast with finite mixtures models, here the distribution of the hidden variable is not bounded to a finite set but is spread out over a given interval. We propose an approach to construct an orthogonal series estimator of the mixing density $f$ involving Legendre polynomials. The construction of the orthonormal sequence varies from one mixture model to another. Minimax upper and lower bounds of the mean integrated squared e…
In this paper we study nonparametric estimators of copulas and copula densities. We first focus our study on a density copula estimator based on a polynomial orthogonal projection of the joint density. A new copula estimator is then deduced. Its asymptotic properties are studied: we provide a large functional class for which this construction is optimal in the minimax and maxiset sense and we propose a method selection for the smoothing parameter. An intensive simulation study shows the very good performance of both copulas and copula densities estimators which we compare to a large panel of c…
In this paper, we study the local constant and the local linear estimators of the conditional density function with right-censored data which exhibit some type of dependence. It is assumed that the observations form a stationary $α-$mixing sequence. The asymptotic normality of the two estimators is established, which combined with the condition that $\lim\limits_{n\rightarrow\infty}nh_nb_n=\infty$ implies the consistency of the two estimators and can be employed to construct confidence intervals for the conditional density function. The result on the local linear estimator of the conditional d…
In this paper, we introduce a new estimator for the emission densities of a nonparametric hidden Markov model. It is adaptive and minimax with respect to each state's regularity--as opposed to globally minimax estimators, which adapt to the worst regularity among the emission densities. Our method is based on Goldenshluger and Lepski's methodology. It is computationally efficient and only requires a family of preliminary estimators, without any restriction on the type of estimators considered. We present two such estimators that allow to reach minimax rates up to a logarithmic term: a spectral…
We study estimation of multivariate densities $p$ of the form $p(x)=h(g(x))$ for $x\in \mathbb {R}^d$ and for a fixed monotone function $h$ and an unknown convex function $g$. The canonical example is $h(y)=e^{-y}$ for $y\in \mathbb {R}$; in this case, the resulting class of densities [\mathcal {P}(e^{-y})={p=\exp(-g):g is convex}] is well known as the class of log-concave densities. Other functions $h$ allow for classes of densities with heavier tails than the log-concave class. We first investigate when the maximum likelihood estimator $\hat{p}$ exists for the class $\mathcal {P}(h)$ for var…
Suppose that $\m{U} = (U_1, \ldots , U_d) $ has a Uniform$([0,1]^d)$ distribution, that $\m{Y} = (Y_1 , \ldots , Y_d) $ has the distribution $G$ on $\RR_+^d$, and let $\m{X} = (X_1 , \ldots , X_d) = (U_1 Y_1 , \ldots , U_d Y_d )$. The resulting class of distributions of $\m{X}$ (as $G$ varies over all distributions on $\RR_+^d$) is called the {\sl Scale Mixture of Uniforms} class of distributions, and the corresponding class of densities on $\RR_+^d$ is denoted by $\{\cal F}_{SMU}(d)$. We study maximum likelihood estimation in the family ${\cal F}_{SMU}(d)$. We prove existence of the MLE, esta…
This paper studies multivariate nonparametric change point localization and inference problems. The data consists of a multivariate time series with potentially short range dependence. The distribution of this data is assumed to be piecewise constant with densities in a Hölder class. The change points, or times at which the distribution changes, are unknown. We derive the limiting distributions of the change point estimators when the minimal jump size vanishes or remains constant, a first in the literature on change point settings. We are introducing two new features: a consistent estimator th…
We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in $d$-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accuracy of every NPMLE. Our results imply, in particular, that every NPMLE achieves near parametric risk (up to logarithmic multiplicative factors) when the true density is a discrete Gaussian mixture without any prior information on the number of mixture components. NPMLEs can natural…
We study nonparametric density estimation problems where error is measured in the Wasserstein distance, a metric on probability distributions popular in many areas of statistics and machine learning. We give the first minimax-optimal rates for this problem for general Wasserstein distances, and show that, unlike classical nonparametric density estimation, these rates depend on whether the densities in question are bounded below. Motivated by variational problems involving the Wasserstein distance, we also show how to construct discretely supported measures, suitable for computational purposes,…
In this article, we consider the estimation of the structural change point in the nonparametric model with dependent observations. We introduce a maximum-CUSUM-estimation procedure, where the CUSUM statistic is constructed based on the sum-of-squares aggregation of the difference of the two Nadaraya-Watson estimates using the observations before and after a specific time point. Under some mild conditions, we prove that the statistic tends to zero almost surely if there is no change, and is larger than a threshold asymptotically almost surely otherwise, which helps us to obtain a threshold-dete…
We consider nonparametric estimation of $L_2$, Renyi-$α$ and Tsallis-$α$ divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of $n^{-1/2}$ when the densities' smoothness, $s$, are both at least $d/4$ where $d$ is the dimension. We also derive minimax lower bounds for this problem which con…
We consider a nonparametric regression model $Y=r(X)+\varepsilon$ with a random covariate $X$ that is independent of the error $\varepsilon$. Then the density of the response $Y$ is a convolution of the densities of $\varepsilon$ and $r(X)$. It can therefore be estimated by a convolution of kernel estimators for these two densities, or more generally by a local von Mises statistic. If the regression function has a nowhere vanishing derivative, then the convolution estimator converges at a parametric rate. We show that the convergence holds uniformly, and that the corresponding process obeys a …
We consider kernel estimation of marginal densities and regression functions of stationary processes. It is shown that for a wide class of time series, with proper centering and scaling, the maximum deviations of kernel density and regression estimates are asymptotically Gumbel. Our results substantially generalize earlier ones which were obtained under independence or beta mixing assumptions. The asymptotic results can be applied to assess patterns of marginal densities or regression functions via the construction of simultaneous confidence bands for which one can perform goodness-of-fit test…
In this paper we consider the nonparametric estimation of density and regression functions with non-negative support using a gamma kernel procedure introduced by Chen (2000). Strong uniform consistency and asymptotic normality of the corresponding estimators are established under a general ergodic assumption on the data generation process. Our results generalize those of Shi and Song (2016), obtained in the classic i.i.d. framework, and the works of Bouezmarni and Rombouts (2008, 2010b) and Gospodinov and Hirukawa (2007) for mixing time series.
Nonparametric density estimators are studied for $d$-dimensional, strongly spatial mixing data which is defined on a general $N$-dimensional lattice structure. We consider linear and nonlinear hard thresholded wavelet estimators which are derived from a $d$-dimensional multiresolution analysis. We give sufficient criteria for the consistency of these estimators and derive rates of convergence in $L^{p'}$ for $p'\in [1,\infty)$. For this reason, we study density functions which are elements of a $d$-dimensional Besov space $B^s_{p,q}(\mathbb{R}^d)$. We also verify the analytic correctness of ou…
The main purpose of this paper is to estimate the regression function by using a recursive nonparametric kernel approach. We derive the asymptotic normality for a general class of recursive kernel estimate of the regression function, under strong mixing conditions. Our purpose is to extend the work of Roussas and Tran [17] concerning the Devroye-Wagner estimate.
Nonparametric estimation of a mixing density based on observations from the corresponding mixture is a challenging statistical problem. This paper surveys the literature on a fast, recursive estimator based on the predictive recursion algorithm. After introducing the algorithm and giving a few examples, I summarize the available asymptotic convergence theory, describe an important semiparametric extension, and highlight two interesting applications. I conclude with a discussion of several recent developments in this area and some open problems.
Nonparametric identification and maximum likelihood estimation for finite-state hidden Markov models are investigated. We obtain identification of the parameters as well as the order of the Markov chain if the transition probability matrices have full-rank and are ergodic, and if the state-dependent distributions are all distinct, but not necessarily linearly independent. Based on this identification result, we develop nonparametric maximum likelihood estimation theory. First, we show that the asymptotic contrast, the Kullback--Leibler divergence of the hidden Markov model, identifies the true…
In this paper we develop a nonparametric maximum likelihood estimate of the mixing distribution of the parameters of a linear stochastic dynamical system. This includes, for example, pharmacokinetic population models with process and measurement noise that are linear in the state vector, input vector and the process and measurement noise vectors. Most research in mixing distributions only considers measurement noise. The advantages of the models with process noise are that, in addition to the measurements errors, the uncertainties in the model itself are taken into the account. For example, fo…
Given a nonparametric Hidden Markov Model (HMM) with two states, the question of constructing efficient multiple testing procedures is considered, treating one of the states as an unknown null hypothesis. A procedure is introduced, based on nonparametric empirical Bayes ideas, that controls the False Discovery Rate (FDR) at a user--specified level. Guarantees on power are also provided, in the form of a control of the true positive rate. One of the key steps in the construction requires supremum--norm convergence of preliminary estimators of the emission densities of the HMM. We provide the ex…
Copula modelling has become ubiquitous in modern statistics. Here, the problem of nonparametrically estimating a copula density is addressed. Arguably the most popular nonparametric density estimator, the kernel estimator is not suitable for the unit-square-supported copula densities, mainly because it is heavily affected by boundary bias issues. In addition, most common copulas admit unbounded densities, and kernel methods are not consistent in that case. In this paper, a kernel-type copula density estimator is proposed. It is based on the idea of transforming the uniform marginals of the cop…
An empirical Bayes problem has an unknown prior to be estimated from data. The predictive recursion (PR) algorithm provides fast nonparametric estimation of mixing distributions and is ideally suited for empirical Bayes applications. This paper presents a general notion of empirical Bayes asymptotic optimality, and it is shown that PR-based procedures satisfy this property under certain conditions. As an application, the problem of in-season prediction of baseball batting averages is considered. There the PR-based empirical Bayes rule performs well in terms of prediction error and ability to c…
This paper deals with nonparametric estimation of conditional den-sities in mixture models in the case when additional covariates are available. The proposed approach consists of performing a prelim-inary clustering algorithm on the additional covariates to guess the mixture component of each observation. Conditional densities of the mixture model are then estimated using kernel density estimates ap-plied separately to each cluster. We investigate the expected L 1 -error of the resulting estimates and derive optimal rates of convergence over classical nonparametric density classes provided the…
We provide necessary and sufficient conditions of uniform consistency of nonparametric sets of alternatives of chi-squared test for testing of hypothesis of homogeneity. The number of cells of chi-squared test increases with sample size growth. Nonparametric sets of alternatives can be defined both in terms of densities and distribution functions.
We review a nonparametric version of Amari's Information Geometry in which the set of positive probability densities on a given sample space is endowed with an atlas of charts to form a differentiable manifold modeled on Orlicz Banach spaces. This nonparametric setting is used to discuss the setting of typical problems in Machine Learning and Statistical Physics, such as relaxed optimization, Kullback-Leibler divergence, Boltzmann entropy, Boltzmann equation
Finite state space hidden Markov models are flexible tools to model phenomena with complex time dependencies: any process distribution can be approximated by a hidden Markov model with enough hidden states.We consider the problem of estimating an unknown process distribution using nonparametric hidden Markov models in the misspecified setting, that is when the data-generating process may not be a hidden Markov model.We show that when the true distribution is exponentially mixing and satisfies a forgetting assumption, the maximum likelihood estimator recovers the best approximation of the true …
Data harmonization is the process by which an equivalence is developed between two variables measuring a common trait. Our problem is motivated by dementia research in which multiple tests are used in practice to measure the same underlying cognitive ability such as language or memory. We connect this statistical problem to mixing distribution estimation. We introduce and study a non-parametric latent trait model, develop a method which enforces uniqueness of the regularized maximum likelihood estimator, show how a nonparametric EM algorithm will converge weakly to its maximizer, and additiona…
We consider a continuous-time stochastic volatility model. The model contains a stationary volatility process, the multivariate density of the finite dimensional distributions of which we aim to estimate. We assume that we observe the process at discrete instants in time. The sampling times will be equidistant with vanishing distance. A multivariate Fourier-type deconvolution kernel density estimator based on the logarithm of the squared processes is proposed to estimate the multivariate volatility density. An expansion of the bias and a bound on the variance are derived. Key words: stochastic…
The differential-geometric structure of the set of positive densities on a given measure space has raised the interest of many mathematicians after the discovery by C.R. Rao of the geometric meaning of the Fisher information. Most of the research is focused on parametric statistical models. In series of papers by author and coworkers a particular version of the nonparametric case has been discussed. It consists of a minimalistic structure modeled according the theory of exponential families: given a reference density other densities are represented by the centered log likelihood which is an el…
We study the multivariate nonparametric change point detection problem, where the data are a sequence of independent $p$-dimensional random vectors whose distributions are piecewise-constant with Lipschitz densities changing at unknown times, called change points. We quantify the size of the distributional change at any change point with the supremum norm of the difference between the corresponding densities. We are concerned with the localization task of estimating the positions of the change points. In our analysis, we allow for the model parameters to vary with the total number of time poin…
We introduce a nonparametric prior on the conditional distribution of a (univariate or multivariate) response given a set of predictors. The prior is constructed in the form of a two-stage generative procedure, which in the first stage recursively partitions the predictor space, and then in the second stage generates the conditional distribution by a multi-scale nonparametric density model on each predictor partition block generated in the first stage. This design allows adaptive smoothing on both the predictor space and the response space, and it results in the full posterior conjugacy of the…
We consider the problem of adaptive estimation of the regression function in a framework where we replace ergodicity assumptions (such as independence or mixing) by another structural assumption on the model. Namely, we propose adaptive upper bounds for kernel estimators with data-driven bandwidth (Lepski's selection rule) in a regression model where the noise is an increment of martingale. It includes, as very particular cases, the usual i.i.d. regression and auto-regressive models. The cornerstone tool for this study is a new result for self-normalized martingales, called ``stability'', whic…
We consider the estimation of densities in multiple subpopulations, where the available sample size in each subpopulation greatly varies. This problem occurs in epidemiology, for example, where different diseases may share similar pathogenic mechanism but differ in their prevalence. Without specifying a parametric form, our proposed method pools information from the population and estimate the density in each subpopulation in a data-driven fashion. Drawing from functional data analysis, low-dimensional approximating density families in the form of exponential families are constructed from the …
Considering multivariate strongly mixing time series, nonparametric tests for a constant copula with specified or unspecified change point (candidate) are derived; the tests are consistent against general alternatives. A tapered block multiplier technique based on serially dependent multiplier random variables is provided to estimate p-values of the test statistics. Size and power of the tests in finite samples are evaluated with Monte Carlo simulations. The block multiplier technique might have several other applications for statistical inference on copulas of serially dependent data.
This paper deals with the nonparametric estimation in heteroscedastic regression $ Y_i=f(X_i)+ξ_i, \: i=1,...,n $, with incomplete information, i.e. each real random variable $ ξ_i $ has a density $ g_{i} $ which is unknown to the statistician. The aim is to estimate the regression function $ f $ at a given point. Using a local polynomial fitting from M-estimator denoted $ \hat f^h $ and applying Lepski's procedure for the bandwidth selection, we construct an estimator $ \hat f^{\hat h} $ which is adaptive over the collection of isotropic Hölder classes. In particular, we establish new exponen…
In this paper, we investigate the asymptotic properties of nonparametric Bayesian mixtures of Betas for estimating a smooth density on $[0,1]$. We consider a parametrization of Beta distributions in terms of mean and scale parameters and construct a mixture of these Betas in the mean parameter, while putting a prior on this scaling parameter. We prove that such Bayesian nonparametric models have good frequentist asymptotic properties. We determine the posterior rate of concentration around the true density and prove that it is the minimax rate of concentration when the true density belongs to …
We study nonparametric isotonic confidence intervals for monotone functions. In Banerjee and Wellner (2001) pointwise confidence intervals, based on likelihood ratio tests for the restricted and unrestricted MLE in the current status model, are introduced. We extend the method to the treatment of other models with monotone functions, and demonstrate our method by a new proof of the results in Banerjee and Wellner (2001) and also by constructing confidence intervals for monotone densities, for which still theory had to be developed. For the latter model we prove that the limit distribution of t…
The problem we concentrate on is as follows: given (1) a convex compact set $X$ in ${\mathbb{R}}^n$, an affine mapping $x\mapsto A(x)$, a parametric family $\{p_μ(\cdot)\}$ of probability densities and (2) $N$ i.i.d. observations of the random variable $ω$, distributed with the density $p_{A(x)}(\cdot)$ for some (unknown) $x\in X$, estimate the value $g^Tx$ of a given linear form at $x$. For several families $\{p_μ(\cdot)\}$ with no additional assumptions on $X$ and $A$, we develop computationally efficient estimation routines which are minimax optimal, within an absolute constant factor. We t…
In this work we deal with the problem of fitting an error density to the goodness-of-fit test of the errors in nonlinear autoregressive time series models with stationary $α$-mixing error terms. The test statistic is based on the integrated squared error of the nonparametric error density estimate and the null error density. By deriving the asymptotic normality of test statistics in these models, we extend the result of Cheng and Sun (Statist. Probab. Lett. \textbf{78}, 1(2008), 50-59) in the model with i.i.d error terms to the more general case.
