Mixture models have received considerable attention recently and Newton [Sankhyā Ser. A 64 (2002) 306--322] proposed a fast recursive algorithm for estimating a mixing distribution. We prove almost sure consistency of this recursive estimate in the weak topology under mild conditions on the family of densities being mixed. This recursive estimate depends on the data ordering and a permutation-invariant modification is proposed, which is an average of the original over permutations of the data sequence. A Rao--Blackwell argument is used to prove consistency in probability of this alternative estimate. Several simulations are presented, comparing the finite-sample performance of the recursive estimate and a Monte Carlo approximation to the permutation-invariant alternative along with that of the nonparametric maximum likelihood estimate and a nonparametric Bayes estimate.
The main purpose of this paper is to estimate the regression function by using a recursive nonparametric kernel approach. We derive the asymptotic normality for a general class of recursive kernel estimate of the regression function, under strong mixing conditions. Our purpose is to extend the work of Roussas and Tran [17] concerning the Devroye-Wagner estimate.
The finite Gamma mixture model is often used to describe randomness in income data, insurance data, and data from other applications. The popular likelihood approach, however, does not work for this model because the likelihood function is unbounded, and the maximum likelihood estimator is therefore not well defined. There has been much research into ways to ensure the consistent estimation of the mixing distribution, including placing an upper bound on the shape parameter or adding a penalty to the log-likelihood function. In this paper, we show that if the shape parameter in the finite Gamma…
A Bayesian framework is attractive in the context of prediction, but a fast recursive update of the predictive distribution has apparently been out of reach, in part because Monte Carlo methods are generally used to compute the predictive. This paper shows that online Bayesian prediction is possible by characterizing the Bayesian predictive update in terms of a bivariate copula, making it unnecessary to pass through the posterior to update the predictive. In standard models, the Bayesian predictive update corresponds to familiar choices of copula but, in nonparametric problems, the appropriate…
Asymptotic theory of tail index estimation has been studied extensively in the frequentist literature on extreme values, but rarely in the Bayesian context. We investigate whether popular Bayesian kernel mixture models are able to support heavy tailed distributions and consistently estimate the tail index. We show that posterior inconsistency in tail index is surprisingly common for both parametric and nonparametric mixture models. We then present a set of sufficient conditions under which posterior consistency in tail index can be achieved, and verify these conditions for Pareto mixture model…
This paper develops nonparametric estimation for discrete choice models based on the mixed multinomial logit (MMNL) model. It has been shown that MMNL models encompass all discrete choice models derived under the assumption of random utility maximization, subject to the identification of an unknown distribution $G$. Noting the mixture model description of the MMNL, we employ a Bayesian nonparametric approach, using nonparametric priors on the unknown mixing distribution $G$, to estimate choice probabilities. We provide an important theoretical support for the use of the proposed methodology by…
We consider the recursive estimation of a regression functional where the explanatory variables take values in some functional space. We prove the almost sure convergence of such estimates for dependent functional data. Also we derive the mean quadratic error of the considered class of estimators. Our results are established with rates and asymptotic appear bounds, under strong mixing condition.
We study the computational complexity of a Metropolis-Hastings algorithm for Bayesian community detection. We first establish a posterior strong consistency result for a natural prior distribution on stochastic block models under the optimal signal-to-noise ratio condition in the literature. We then give a set of conditions that guarantee rapid mixing of a simple Metropolis-Hastings algorithm. The mixing time analysis is based on a careful study of posterior ratios and a canonical path argument to control the spectral gap of the Markov chain.
A family of Markov blankets in a faithful Bayesian network satisfies the symmetry and consistency properties. In this paper, we draw a bijection between families of consistent Markov blankets and moral graphs. We define the new concepts of weak recursive simpliciality and perfect elimination kits. We prove that they are equivalent to graph morality. In addition, we prove that morality can be decided in polynomial time for graphs with maximum degree less than $5$, but the problem is NP-complete for graphs with higher maximum degrees.
We study the computational complexity of Markov chain Monte Carlo (MCMC) methods for high-dimensional Bayesian linear regression under sparsity constraints. We first show that a Bayesian approach can achieve variable-selection consistency under relatively mild conditions on the design matrix. We then demonstrate that the statistical criterion of posterior concentration need not imply the computational desideratum of rapid mixing of the MCMC algorithm. By introducing a truncated sparsity prior for variable selection, we provide a set of conditions that guarantee both variable-selection consiste…
Many statistical problems involve mixture models and the need for computationally efficient methods to estimate the mixing distribution has increased dramatically in recent years. Newton [Sankhya Ser. A 64 (2002) 306--322] proposed a fast recursive algorithm for estimating the mixing distribution, which we study as a special case of stochastic approximation (SA). We begin with a review of SA, some recent statistical applications, and the theory necessary for analysis of a SA algorithm, which includes Lyapunov functions and ODE stability theory. Then standard SA results are used to prove consis…
Estimating the mixing density of a mixture distribution remains an interesting problem in statistics literature. Using a stochastic approximation method, Newton and Zhang (1999) introduced a fast recursive algorithm for estimating the mixing density of a mixture. Under suitably chosen weights the stochastic approximation estimator converges to the true solution. In Tokdar et. al. (2009) the consistency of this recursive estimation method was established. However, the proof of consistency of the resulting estimator used independence among observations as an assumption. Here, we extend the inves…
This paper outlines a new nonparametric estimation procedure for unobserved phi-mixing processes. It is assumed that the only information on the stationary hidden states (Xk) is given by the process (Yk), where Yk is a noisy observation of f(Xk). The paper introduces a maximum pseudo-likelihood procedure to estimate the function f and the distribution of the hidden states using blocks of observations of length b. The identifiability of the model is studied in the particular cases b=1 and b=2. The consistency of the estimators of f and of the distribution of the hidden states as the number of o…
We consider nonparametric estimation for functional autoregressive processes with Markov switching. First, we study the case where complete data is available; i.e. when we observe the Markov switching regime. Then we estimate the regression function in each regime using a Nadaraya-Watson type estimator. Second, we introduce a nonparametric recursive algorithm in the case of hidden Markov switching regime. Our algorithm restores the missing data by means of a Monte-Carlo step and estimate the regression function via a Robbins-Monro step. Consistency of the estimators are proved in both cases. F…
In this paper, we study the behavior of a kernel estimator of the regression function in the right censored model with $α$-mixing data . The uniform strong consistency over a real compact set of the estimate is established along with a rate of convergence. Some simulations are carried out to illustrate the behavior of the estimate with different examples for finite sample sizes.
We study a nonparametric regression model for sample data which is defined on an $N$-dimensional lattice structure and which is assumed to be strong spatial mixing: we use design adapted multidimensional Haar wavelets which form an orthonormal system w.r.t. the empirical measure of the sample data. For such orthonormal systems, we consider a nonparametric hard thresholding estimator. We give sufficient criteria for the consistency of this estimator and we derive rates of convergence. The theorems reveal that our estimator is able to adapt to the local smoothness of the underlying regression fu…
We consider the problem of forecasting the next (observable) state of an unknown ergodic dynamical system from a noisy observation of the present state. Our main result shows, for example, that support vector machines (SVMs) using Gaussian RBF kernels can learn the best forecaster from a sequence of noisy observations if (a) the unknown observational noise process is bounded and has a summable $α$-mixing rate and (b) the unknown ergodic dynamical system is defined by a Lipschitz continuous function on some compact subset of $\mathbb{R}^d$ and has a summable decay of correlations for Lipschitz …
The validity of various bootstrapping methods has been proved for the sample mean of strongly mixing data. But in many applications, there appear nonlinear statistics of processes that are not strongly mixing. We investigate the nonoverlapping block bootstrap sequences which are near epoch dependent on strong mixing or absolutely regular processes. This includes ARMA and GARCH-processes as well as data from chaotic dynamical systems. We establish the strong consistency of the bootstrap distribution estimator not only for the sample mean, but also for U-statistics, which include examples as Gin…
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学数学与计算机科学学院）；Vietnam National University Ho Chi Minh City, Vietnam（越南胡志明市国家大学）；Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（越南胡志明市科学大学信息技术学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析 excellence 中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane City, Australia（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
