Estimation of the mixing distribution under a general mixture model is a very difficult problem, especially when the mixing distribution is assumed to have a density. Predictive recursion (PR) is a fast, recursive algorithm for nonparametric estimation of a mixing distribution/density in general mixture models. However, the existing PR consistency results make rather strong assumptions, some of which fail for a class of mixture models relevant for monotone density estimation, namely, scale mixtures of uniform kernels. In this paper, we develop new consistency results for PR under weaker conditions. Armed with this new theory, we prove that PR is consistent for the scale mixture of uniforms problem, and we show that the corresponding PR mixture density estimator has very good practical performance compared to several existing methods for monotone density estimation.
In this paper we prove large and moderate deviations principles for the recursive kernel estimator of a probability density function and its partial derivatives. Unlike the density estimator, the derivatives estimators exhibit a quadratic behavior not only for the moderate deviations scale but also for the large deviations one. We provide results both for the pointwise and the uniform deviations.
In this paper we propose an automatic bandwidth selection of the recursive kernel density estimators with missing data in the context of global and local density estimation. We showed that, using the selected bandwidth and a special stepsize, the proposed recursive estimators outperformed the nonrecursive one in terms of estimation error in the case of global estimation. However, the recursive estimators are much better in terms of computational costs. We corroborated these theoretical results through simulation studies and on the simulated data of the Aquitaine cohort of HIV-1 infected patien…
The main purpose of this paper is to estimate the regression function by using a recursive nonparametric kernel approach. We derive the asymptotic normality for a general class of recursive kernel estimate of the regression function, under strong mixing conditions. Our purpose is to extend the work of Roussas and Tran [17] concerning the Devroye-Wagner estimate.
In this paper we prove large and moderate deviations principles for the recursive kernel estimators of a probability density function defined by the stochastic approximation algorithm introduced by Mokkadem et al. [2009. The stochastic approximation method for the estimation of a probability density. J. Statist. Plann. Inference 139, 2459-2478]. We show that the estimator constructed using the stepsize which minimize the variance of the class of the recursive estimators defined in Mokkadem et al. (2009) gives the same pointwise LDP and MDP as the Rosenblatt kernel estimator. We provide results…
In this paper we propose an automatic selection of the bandwidth of the semi-recursive kernel estimators of a regression function defined by the stochastic approximation algorithm. We showed that, using the selected bandwidth and some special stepsizes, the proposed semi-recursive estimators will be very competitive to the nonrecursive one in terms of estimation error but much better in terms of computational costs. We corroborated these theoretical results through simulation study and a real dataset.
In a pioneer work, Révész (1973) introduces the stochastic approximation method to build up a recursive kernel estimator of the regression function $x\mapsto E(Y|X=x)$. However, according to Révész (1977), his estimator has two main drawbacks: on the one hand, its convergence rate is smaller than that of the nonrecursive Nadaraya-Watson's kernel regression estimator, and, on the other hand, the required assumptions on the density of the random variable $X$ are stronger than those usually needed in the framework of regression estimation. We first come back on the study of the convergence rate o…
We investigate the asymptotic behavior of the Nadaraya-Watson estimator for the estimation of the regression function in a semiparametric regression model. On the one hand, we make use of the recursive version of the sliced inverse regression method for the estimation of the unknown parameter of the model. On the other hand, we implement a recursive Nadaraya-Watson procedure for the estimation of the regression function which takes into account the previous estimation of the parameter of the semiparametric regression model. We establish the almost sure convergence as well as the asymptotic nor…
In this paper, we investigate a nonparametric approach to provide a recursive estimator of the transition density of a non-stationary piecewise-deterministic Markov process, from only one observation of the path within a long time. In this framework, we do not observe a Markov chain with transition kernel of interest. Fortunately, one may write the transition density of interest as the ratio of the invariant distributions of two embedded chains of the process. Our method consists in estimating these invariant measures. We state a result of consistency and a central limit theorem under some gen…
We apply the stochastic approximation method to construct a large class of recursive kernel estimators of a probability density, including the one introduced by Hall and Patil (1994). We study the properties of these estimators and compare them with Rosenblatt's nonrecursive estimator. It turns out that, for pointwise estimation, it is preferable to use the nonrecursive Rosenblatt's kernel estimator rather than any recursive estimator. A contrario, for estimation by confidence intervals, it is better to use a recursive estimator rather than Rosenblatt's estimator.
The Neyman and Scott (1948) model is widely used to demonstrate a serious weakness of the Maximum Likelihood (ML) method: it can give rise to inconsistent estimators. The primary objective of this paper is to revisit this example with a view to demonstrate that the culprit for the inconsistent estimation is not the ML method but an ill-defined statistical model. It is also shown that a simple recasting of this model renders it well-defined and the ML method gives rise to consistent and asymptotically efficient estimators.
In this paper we propose a recursive online algorithm for estimating the parameters of a time-varying ARCH process. The estimation is done by updating the estimator at time point $t-1$ with observations about the time point $t$ to yield an estimator of the parameter at time point $t$. The sampling properties of this estimator are studied in a non-stationary context -- in particular, asymptotic normality and an expression for the bias due to non-stationarity are established. By running two recursive online algorithms in parallel with different step sizes and taking a linear combination of the e…
Assuming that one-step transition kernel of a discrete time, time-homogenous Markov chain model is parameterized by a parameter $θ\in \boldsymbol Θ$, we derive a recursive (in time) construction of confidence regions for the unknown parameter of interest, say $θ^*\in \boldsymbol Θ$. It is supposed that the observed data used in construction of the confidence regions is generated by a Markov chain whose transition kernel corresponds to $θ^*$ . The key step in our construction is derivation of a recursive scheme for an appropriate point estimator of $θ^*$. To achieve this, we start by what we ca…
In this paper, we prove large deviations principle for the Nadaraya-Watson estimator and for the semi-recursive kernel estimator of the regression in the multidimensional case. Under suitable conditions, we show that the rate function is a good rate function. We thus generalize the results already obtained in the unidimensional case for the Nadaraya-Watson estimator. Moreover, we give a moderate deviations principle for these two estimators. It turns out that the rate function obtained in the moderate deviations principle for the semi-recursive estimator is larger than the one obtained for the…
The paper deals with the statistical analysis of several data sets associated with shape invariant models with different translation, height and scaling parameters. We propose to estimate these parameters together with the common shape function. Our approach extends the recent work of Bercu and Fraysse to multivariate shape invariant models. We propose a very efficient Robbins-Monro procedure for the estimation of the translation parameters and we use these estimates in order to evaluate scale parameters. The main pattern is estimated by a weighted Nadaraya-Watson recursive estimator. We provi…
We address the identifiablity and estimation of recursive max-linear structural equation models represented by an edge weighted directed acyclic graph (DAG). Such models are generally unidentifiable and we identify the whole class of DAGs and edge weights corresponding to a given observational distribution. For estimation, standard likelihood theory cannot be applied because the corresponding families of distributions are not dominated. Given the underlying DAG, we present an estimator for the class of edge weights and show that it can be considered a generalized maximum likelihood estimator. …
The main purpose is to estimate the regression function of a real random variable with functional explanatory variable by using a recursive nonparametric kernel approach. The mean square error and the almost sure convergence of a family of recursive kernel estimates of the regression function are derived. These results are established with rates and precise evaluation of the constant terms. Also, a central limit theorem for this class of estimators is established. The method is evaluated on simulations and real data set studies.
This paper focuses on the problem of recursive nonlinear least squares parameter estimation in multi-agent networks, in which the individual agents observe sequentially over time an independent and identically distributed (i.i.d.) time-series consisting of a nonlinear function of the true but unknown parameter corrupted by noise. A distributed recursive estimator of the \emph{consensus} + \emph{innovations} type, namely $\mathcal{CIWNLS}$, is proposed, in which the agents update their parameter estimates at each observation sampling epoch in a collaborative way by simultaneously processing the…
In this work, we study a new recursive stochastic algorithm for the joint estimation of quantile and superquantile of an unknown distribution. The novelty of this algorithm is to use the Cesaro averaging of the quantile estimation inside the recursive approximation of the superquantile. We provide some sharp non-asymptotic bounds on the quadratic risk of the superquantile estimator for different step size sequences. We also prove new non-asymptotic $L^p$-controls on the Robbins Monro algorithm for quantile estimation and its averaged version. Finally, we derive a central limit theorem of our j…
This paper considers the asymptotic properties of the recursive maximum likelihood estimation in hidden Markov models. The paper is focused on the asymptotic behavior of the log-likelihood function and on the point-convergence and convergence rate of the recursive maximum likelihood estimator. Using the principle of analytical continuation, the analyticity of the asymptotic log-likelihood function is shown for analytically parameterized hidden Markov models. Relying on this fact and some results from differential geometry (Lojasiewicz inequality), the almost sure point-convergence of the recur…
We consider estimation of the cumulative incidence function (CIF) in the competing risks Cox model. We study three methods. Methods 1 and 2 are existing methods while Method 3 is a newly-proposed method. Method 3 is constructed so that the sum of the CIF's across all event types at the last observed event time is guaranteed, assuming no ties, to be equal to 1. The performance of the methods is examined in a simulation study, and the methods are illustrated on a data example from the field of computer code comprehension. The newly-proposed Method 3 exhibits performance comparable to that of Met…
A recursive estimator of the conditional geometric median in Hilbert spaces is studied. It is based on a stochastic gradient algorithm whose aim is to minimize a weighted L1 criterion and is consequently well adapted for robust online estimation. The weights are controlled by a kernel function and an associated bandwidth. Almost sure convergence and L2 rates of convergence are proved under general conditions on the conditional distribution as well as the sequence of descent steps of the algorithm and the sequence of bandwidths. Asymptotic normality is also proved for the averaged version of th…
Bivariate Frank Copula: Some More Results on Point Estimation of the Association Parameter from a Bayesian Perspective and Revisiting the Goodness of Fit Tests with an Application to Model Groundwater Data from Dong Thap, Vietnam
Faculty of Applied Science, Ho Chi Minh City University of Technology (HCMUT)（胡志明市理工大学应用科学学院）；Vietnam National University Ho Chi Minh City（越南国家大学胡志明市分校）；Faculty of Mathematics and Statistics, Ton Duc Thang University（Ton Duc Thang大学数学与统计学院）
In this paper we prove large deviations principles for the averaged stochastic approximation method for the estimation of a regression function introduced by A. Mokkadem et al. [Revisiting Révész's stochastic approximation method for the estimation of a regression function, ALEA Lat. Amm. J. Probab. Math. Stat. 6 (2009), 63-114]. We show that the averaged stochastic approximation algorithm constructed using the weight sequence which minimize the asymptotic variance gives the same pointwise LDP as the Nadaraya-Watson kernel estimator. Moreover, we give a moderate deviations principle for these …
This paper describes a recursive estimation procedure for multivariate binary densities (probability distributions of vectors of Bernoulli random variables) using orthogonal expansions. For $d$ covariates, there are $2^d$ basis coefficients to estimate, which renders conventional approaches computationally prohibitive when $d$ is large. However, for a wide class of densities that satisfy a certain sparsity condition, our estimator runs in probabilistic polynomial time and adapts to the unknown sparsity of the underlying density in two key ways: (1) it attains near-minimax mean-squared error fo…
This paper introduces a family of recursively defined estimators of the parameters of a diffusion process. We use ideas of stochastic algorithms for the construction of the estimators. Asymptotic consistency of these estimators and asymptotic normality of an appropriate normalization are proved. The results are applied to two examples from the financial literature; viz., Cox-Ingersoll-Ross' model and the constant elasticity of variance (CEV) process illustrate the use of the technique proposed herein.
A piecewise-deterministic Markov process is a stochastic process whose behavior is governed by an ordinary differential equation punctuated by random jumps occurring at random times. We focus on the nonparametric estimation problem of the jump rate for such a stochastic model observed within a long time interval under an ergodicity condition. We introduce an uncountable class (indexed by the deterministic flow) of recursive kernel estimates of the jump rate and we establish their strong pointwise consistency as well as their asymptotic normality. We propose to choose among this class the estim…
When large amounts of data continuously arrive in streams, online updating is an effective way to reduce storage and computational burden. The key idea of online updating is that the previous estimators are sequentially updated only using the current data and some summary statistics of historical raw data. In this article, we develop a renewable learning method for a multiplicative regression model with streaming data, where the parameter estimator based on a least product relative error criterion is renewed without revisiting any historical raw data. Under some regularity conditions, we estab…
In this paper, we consider the alleviation of the boundary problem when the probability density function has bounded support. We apply Robbins-Monro's algorithm and Bernstein polynomials to construct a recursive density estimator. We study the asymptotic properties of the proposed recursive estimator. We then compared our proposed recursive estimator with many others estimators. Finally, we confirm our theoretical result through a simulation study and then using two real datasets.
This paper revisits the work of Rauch et al. (1965) and develops a novel method for recursive maximum likelihood particle filtering for general state-space models. The new method is based on statistical analysis of incomplete observations of the systems. Score function and conditional observed information of the incomplete observations/data are introduced and their distributional properties are discussed. Some identities concerning the score function and information matrices of the incomplete data are derived. Maximum likelihood estimation of state-vector is presented in terms of the score fun…
We consider a recursive algorithm to construct an aggregated estimator from a finite number of base decision rules in the classification problem. The estimator approximately minimizes a convex risk functional under the l1-constraint. It is defined by a stochastic version of the mirror descent algorithm (i.e., of the method which performs gradient descent in the dual space) with an additional averaging. The main result of the paper is an upper bound for the expected accuracy of the proposed estimator. This bound is of the order $\sqrt{(\log M)/t}$ with an explicit and small constant factor, whe…
We consider estimation procedures which are recursive in the sense that each successive estimator is obtained from the previous one by a simple adjustment. The model considered in the paper is very general as we do not impose any preliminary restrictions on the probabilistic nature of the observation process and cover a wide class of nonlinear recursive procedures. In this paper we study asymptotic behaviour of the recursive estimators. The results of the paper can be used to determine the form of a recursive procedure which is expected to have the same asymptotic properties as the correspondi…
This paper considers the problem of cardinality estimation in data stream applications. We present a statistical analysis of probabilistic counting algorithms, focusing on two techniques that use pseudo-random variates to form low-dimensional data sketches. We apply conventional statistical methods to compare probabilistic algorithms based on storing either selected order statistics, or random projections. We derive estimators of the cardinality in both cases, and show that the maximal-term estimator is recursively computable and has exponentially decreasing error bounds. Furthermore, we show …
We consider the estimation of dynamic discrete choice models in a semiparametric setting, in which the per-period utility functions are known up to a finite number of parameters, but the distribution of utility shocks is left unspecified. This semiparametric setup differs from most of the existing identification and estimation literature for dynamic discrete choice mod- els. To show identification we derive and exploit a new Bellman-like recursive representation for the unknown quantile function of the utility shocks. Our estimators are straightforward to compute; some are simple and require n…
We consider estimation procedures which are recursive in the sense that each successive estimator is obtained from the previous one by a simple adjustment. We study rate of convergence of recursive estimation procedures for the general statistical model.
We consider estimation procedures which are recursive in the sense that each successive estimator is obtained from the previous one by a simple adjustment. We propose a wide class of recursive estimation procedures for the general statistical model and study convergence.
Maximum-a-posteriori (MAP) estimation is the main Bayesian estimation methodology in imaging sciences, where high dimensionality is often addressed by using Bayesian models that are log-concave and whose posterior mode can be computed efficiently by convex optimisation. Despite its success and wide adoption, MAP estimation is not theoretically well understood yet. The prevalent view in the community is that MAP estimation is not proper Bayesian estimation in a decision-theoretic sense because it does not minimise a meaningful expected loss function (unlike the minimum mean squared error (MMSE)…
We consider the recursive estimation of a regression functional where the explanatory variables take values in some functional space. We prove the almost sure convergence of such estimates for dependent functional data. Also we derive the mean quadratic error of the considered class of estimators. Our results are established with rates and asymptotic appear bounds, under strong mixing condition.
Statistical-Computational Trade-offs for Recursive Adaptive Partitioning Estimators
Department of Statistics and Data Science, National University of Singapore（统计与数据科学系，新加坡国立大学）；Department of Operations Research and Financial Engineering, Princeton University（运筹学与金融工程系，普林斯顿大学）；Department of Statistics, University of California, Davis（统计系，加州大学戴维斯分校）
Recursive linear structural equation models are widely used to postulate causal mechanisms underlying observational data. In these models, each variable equals a linear combination of a subset of the remaining variables plus an error term. When there is no unobserved confounding or selection bias, the error terms are assumed to be independent. We consider estimating a total causal effect in this setting. The causal structure is assumed to be known only up to a maximally oriented partially directed acyclic graph (MPDAG), a general class of graphs that can represent a Markov equivalence class of…
The paper is devoted to the study of a parametric deformation model of independent and identically random variables. Firstly, we construct an efficient and very easy to compute recursive estimate of the parameter. Our stochastic estimator is similar to the Robbins-Monro procedure where the contrast function is the Wasserstein distance. Secondly, we propose a recursive estimator similar to that of Parzen-Rosenblatt kernel density estimator in order to estimate the density of the random variables. This estimate takes into account the previous estimation of the parameter of the model. Finally, we…
A new matching method is proposed for the estimation of the average treatment effect of social policy interventions (e.g., training programs or health care measures). Given an outcome variable, a treatment and a set of pre-treatment covariates, the method is based on the examination of random recursive partitions of the space of covariates using regression trees. A regression tree is grown either on the treated or on the untreated individuals {\it only} using as response variable a random permutation of the indexes 1...$n$ ($n$ being the number of units involved), while the indexes for the oth…
