This article establishes general conditions for posterior consistency of Bayesian finite mixture models with a prior on the number of components. That is, we provide sufficient conditions under which the posterior concentrates on neighborhoods of the true parameter values when the data are generated from a finite mixture over the assumed family of component distributions. Specifically, we establish almost sure consistency for the number of components, the mixture weights, and the component parameters, up to a permutation of the component labels. The approach taken here is based on Doob's theorem, which has the advantage of holding under extraordinarily general conditions, and the disadvantage of only guaranteeing consistency at a set of parameter values that has probability one under the prior. However, we show that in fact, for commonly used choices of prior, this yields consistency at Lebesgue-almost all parameter values -- which is satisfactory for most practical purposes. We aim to formulate the results in a way that maximizes clarity, generality, and ease of use.
Asymptotic theory of tail index estimation has been studied extensively in the frequentist literature on extreme values, but rarely in the Bayesian context. We investigate whether popular Bayesian kernel mixture models are able to support heavy tailed distributions and consistently estimate the tail index. We show that posterior inconsistency in tail index is surprisingly common for both parametric and nonparametric mixture models. We then present a set of sufficient conditions under which posterior consistency in tail index can be achieved, and verify these conditions for Pareto mixture model…
Robust estimation under multivariate normal (MVN) mixture model is always a computational challenge. A recently proposed maximum pseudo \b{eta}-likelihood estimator aims to estimate the unknown parameters of a MVN mixture model in the spirit of minimum density power divergence (DPD) methodology but with a relatively simpler and tractable computational algorithm even for larger dimensions. In this letter, we will rigorously derive the existence and weak consistency of the maximum pseudo \b{eta}-likelihood estimator in case of MVN mixture models under a reasonable set of assumptions.
The finite Gamma mixture model is often used to describe randomness in income data, insurance data, and data from other applications. The popular likelihood approach, however, does not work for this model because the likelihood function is unbounded, and the maximum likelihood estimator is therefore not well defined. There has been much research into ways to ensure the consistent estimation of the mixing distribution, including placing an upper bound on the shape parameter or adding a penalty to the log-likelihood function. In this paper, we show that if the shape parameter in the finite Gamma…
We study uniform consistency in nonparametric mixture models as well as closely related mixture of regression (also known as mixed regression) models, where the regression functions are allowed to be nonparametric and the error distributions are assumed to be convolutions of a Gaussian density. We construct uniformly consistent estimators under general conditions while simultaneously highlighting several pain points in extending existing pointwise consistency results to uniform results. The resulting analysis turns out to be nontrivial, and several novel technical tools are developed along the…
Dirichlet process mixtures are flexible non-parametric models, particularly suited to density estimation and probabilistic clustering. In this work we study the posterior distribution induced by Dirichlet process mixtures as the sample size increases, and more specifically focus on consistency for the unknown number of clusters when the observed data are generated from a finite mixture. Crucially, we consider the situation where a prior is placed on the concentration parameter of the underlying Dirichlet process. Previous findings in the literature suggest that Dirichlet process mixtures are t…
We study location-scale mixture priors for nonparametric statistical problems, including multivariate regression, density estimation and classification. We show that a rate-adaptive procedure can be obtained if the prior is properly constructed. In particular, we show that adaptation is achieved if a kernel mixture prior on a regression function is constructed using a Gaussian kernel, an inverse gamma bandwidth, and Gaussian mixing weights.
Mixture models are widely used in Bayesian statistics and machine learning, in particular in computational biology, natural language processing and many other fields. Variational inference, a technique for approximating intractable posteriors thanks to optimization algorithms, is extremely popular in practice when dealing with complex models such as mixtures. The contribution of this paper is two-fold. First, we study the concentration of variational approximations of posteriors, which is still an open problem for general mixtures, and we derive consistency and rates of convergence. We also ta…
In recent years, multiple hypothesis testing has come to the forefront of statistical research, ostensibly in relation to applications in genomics and some other emerging fields. The false discovery rate (FDR) and its variants provide very important notions of errors in this context comparable to the role of error probabilities in classical testing problems. Accurate estimation of positive FDR (pFDR), a variant of the FDR, is essential in assessing and controlling this measure. In a recent paper, the authors proposed a model-based nonparametric Bayesian method of estimation of the pFDR functio…
Density estimation represents one of the most successful applications of Bayesian nonparametrics. In particular, Dirichlet process mixtures of normals are the gold standard for density estimation and their asymptotic properties have been studied extensively, especially in the univariate case. However a gap between practitioners and the current theoretical literature is present. So far, posterior asymptotic results in the multivariate case are available only for location mixtures of Gaussian kernels with independent prior on the common covariance matrix, while in practice as well as from a conc…
Delattre et al. (2013) considered a system of stochastic differential equations (SDEs) in a random effects setup. Under the independent and identical (iid) situation, and assuming normal distribution of the random effects, they established weak consistency of the maximum likelihood estimators (M LEs) of the population parameters of the random effects. In this article, respecting the increasing importance and versatility of normal mixtures and their ability to approximate any standard distribution, we consider the random effects having mixture of normal distributions and prove asymptotic result…
Brown's 1971 paper "Admissible estimators, recurrent diffusions and insoluble boundary value problems" is a landmark in the admissibility literature. It nearly completely settles the issue of admissibility/inadmissibility for estimating the mean of a multivariate normal distribution with identity covariance under sum of squared error loss. We revisit this wonderful tour de force on its 50th anniversary and present an alternative and more direct proof of the result for generalized Bayes estimators corresponding to priors which are a subclass of scale mixtures of spherical normals.
While Jeffreys priors usually are well-defined for the parameters of mixtures of distributions, they are not available in closed form. Furthermore, they often are improper priors. Hence, they have never been used to draw inference on the mixture parameters. The implementation and the properties of Jeffreys priors in several mixture settings are studied. It is shown that the associated posterior distributions most often are improper. Nevertheless, the Jeffreys prior for the mixture weights conditionally on the parameters of the mixture components will be shown to have the property of conservati…
We show that rate-adaptive multivariate density estimation can be performed using Bayesian methods based on Dirichlet mixtures of normal kernels with a prior distribution on the kernel's covariance matrix parameter. We derive sufficient conditions on the prior specification that guarantee convergence to a true density at a rate that is optimal minimax for the smoothness class to which the true density belongs. No prior knowledge of smoothness is assumed. The sufficient conditions are shown to hold for the Dirichlet location mixture of normals prior with a Gaussian base measure and an inverse-W…
We consider the problem of Bayesian density estimation on the positive semiline for possibly unbounded densities. We propose a hierarchical Bayesian estimator based on the gamma mixture prior which can be viewed as a location mixture. We study convergence rates of Bayesian density estimators based on such mixtures. We construct approximations of the local Hölder densities, and of their extension to unbounded densities, to be continuous mixtures of gamma distributions, leading to approximations of such densities by finite mixtures. These results are then used to derive posterior concentration r…
Finite mixture and Markov-switching models generalize and, therefore, nest specifications featuring only one component. While specifying priors in the two: the general (mixture) model and its special (single-component) case, it may be desirable to ensure that the prior assumptions introduced into both structures are coherent in the sense that the prior distribution in the nested model amounts to the conditional prior in the mixture model under relevant parametric restriction. The study provides the rudiments of setting coherent priors in Bayesian univariate finite mixture and Markov-switching …
Modifications of the BIC for order selection in finite mixture models
La Trobe University（拉 Trobe 大学）；Institute of Mathematics for Industry, Kyushu University（九州大学产业数学研究所）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（ARC 细胞系统数学分析卓越中心）；Queensland University of Technology（昆士兰科技大学）
Exact Consistency Tests for Gaussian Mixture Filters using Normalized Deviation Squared Statistics
Smead Aerospace Engineering Sciences, University of Colorado Boulder（科罗拉多大学博尔德分校斯米德航空航天工程科学学院）；Aurora Flight Sciences, a Boeing Company（波音公司旗下极光飞行科学公司）
Positivity of the prior probability of Kullback-Leibler neighborhood around the true density, commonly known as the Kullback-Leibler property, plays a fundamental role in posterior consistency. A popular prior for Bayesian estimation is given by a Dirichlet mixture, where the kernels are chosen depending on the sample space and the class of densities to be estimated. The Kullback-Leibler property of the Dirichlet mixture prior has been shown for some special kernels like the normal density or Bernstein polynomial, under appropriate conditions. In this paper, we obtain easily verifiable suffici…
In the problem of selecting variables in a multivariate linear regression model, we derive new Bayesian information criteria based on a prior mixing a smooth distribution and a delta distribution. Each of them can be interpreted as a fusion of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Inheriting their asymptotic properties, our information criteria are consistent in variable selection in both the large-sample and the high-dimensional asymptotic frameworks. In numerical simulations, variable selection methods based on our information criteria choose th…
Frequentist conditions for asymptotic suitability of Bayesian procedures focus on lower bounds for prior mass in Kullback-Leibler neighbourhoods of the data distribution. The goal of this paper is to investigate the flexibility in criteria for posterior consistency with i.i.d. data. We formulate a versatile posterior consistency theorem that applies both to well- and mis-specified models and which we use to re-derive Schwartz's theorem, consider Kullback-Leibler consistency and formulate consistency theorems in which priors charge metric balls. It is generalized to sieved models with Barron's …
We examine necessary and sufficient conditions for posterior consistency under $g$-priors, including extensions to hierarchical and empirical Bayesian models. The key features of this article are that we allow the number of regressors to grow at the same rate as the sample size and define posterior consistency under the sup vector norm instead of the more conventional Euclidean norm. We consider in particular the empirical Bayesian model of George and Foster (2000), the hyper-$g$-prior of Liang et al. (2008), and the prior considered by Zellner and Siow (1980).
We study the rates of convergence of the posterior distribution for Bayesian density estimation with Dirichlet mixtures of normal distributions as the prior. The true density is assumed to be twice continuously differentiable. The bandwidth is given a sequence of priors which is obtained by scaling a single prior by an appropriate order. In order to handle this problem, we derive a new general rate theorem by considering a countable covering of the parameter space whose prior probabilities satisfy a summability condition together with certain individual bounds on the Hellinger metric entropy. …
In this paper, we prove almost surely consistency of a Survival Analysis model, which puts a Gaussian process, mapped to the unit interval, as a prior on the so-called hazard function. We assume our data is given by survival lifetimes $T$ belonging to $\mathbb{R}^{+}$, and covariates on $[0,1]^d$, where $d$ is an arbitrary dimension. We define an appropriate metric for survival functions and prove posterior consistency with respect to this metric. Our proof is based on an extension of the theorem of Schwartz (1965), which gives general conditions for proving almost surely consistency in the se…
Mixture models are well-known for their versatility, and the Bayesian paradigm is a suitable platform for mixture analysis, particularly when the number of components is unknown. Bhattacharya (2008) introduced a mixture model based on the Dirichlet process, where an upper bound on the unknown number of components is to be specified. Here we consider a Bayesian asymptotic framework for objectively specifying the upper bound, which we assume to depend on the sample size. In particular, we define a Bayesian analogue of the mean integrated squared error (Bayesian MISE), and select that form of the…
We study posterior contraction behaviors for parameters of interest in the context of Bayesian mixture modeling, where the number of mixing components is unknown while the model itself may or may not be correctly specified. Two representative types of prior specification will be considered: one requires explicitly a prior distribution on the number of mixture components, while the other places a nonparametric prior on the space of mixing distributions. The former is shown to yield an optimal rate of posterior contraction on the model parameters under minimal conditions, while the latter can be…
Estimation of the covariance matrix for high-dimensional multivariate datasets is a challenging and important problem in modern statistics. In this paper, we focus on high-dimensional Gaussian DAG models where sparsity is induced on the Cholesky factor L of the inverse covariance matrix. In recent work, ([Cao, Khare, and Ghosh, 2019]), we established high-dimensional sparsity selection consistency for a hierarchical Bayesian DAG model, where an Erdos-Renyi prior is placed on the sparsity pattern in the Cholesky factor L, and a DAG-Wishart prior is placed on the resulting non-zero Cholesky entr…
Statistical network analysis primarily focuses on inferring the parameters of an observed network. In many applications, especially in the social sciences, the observed data is the groups formed by individual subjects. In these applications, the network is itself a parameter of a statistical model. Zhao and Weko (2019) propose a model-based approach, called the hub model, to infer implicit networks from grouping behavior. The hub model assumes that each member of the group is brought together by a member of the group called the hub. The hub model belongs to the family of Bernoulli mixture mode…
Posterior distribution over a countable set M of continuous data-sampling distributions piles up at L-projection of the true distribution r on M, provided that the L-projection is unique. If there are several L-projections of r on M, then the posterior probability splits among them equally.
While Jeffreys priors usually are well-defined for the parameters of mixtures of distributions, they are not available in closed form. Furthermore, they often are improper priors. Hence, they have never been used to draw inference on the mixture parameters. We study in this paper the implementation and the properties of Jeffreys priors in several mixture settings, show that the associated posterior distributions most often are improper, and then propose a noninformative alternative for the analysis of mixtures.
We investigate the asymptotic behavior of posterior distributions of regression coefficients in high-dimensional linear models as the number of dimensions grows with the number of observations. We show that the posterior distribution concentrates in neighborhoods of the true parameter under simple sufficient conditions. These conditions hold under popular shrinkage priors given some sparsity assumptions.
In many contexts Gaussian Mixtures (GM) are used to approximate probability distributions, possibly time-varying. In some applications the number of GM components exponentially increases over time, and reduction procedures are required to keep them reasonably limited. The GM reduction (GMR) problem can be formulated by choosing different measures of the dissimilarity of GMs before and after reduction, like the Kullback-Leibler Divergence (KLD) and the Integral Squared Error (ISE). Since in no case the solution is obtained in closed form, many approximate GMR algorithms have been proposed in th…
This paper deals with Bayesian inference of a mixture of Gaussian distributions. A novel formulation of the mixture model is introduced, which includes the prior constraint that each Gaussian component is always assigned a minimal number of data points. This enables noninformative improper priors such as the Jeffreys prior to be placed on the component parameters. We demonstrate difficulties involved in specifying a prior for the standard Gaussian mixture model, and show how the new model can be used to overcome these. MCMC methods are given for efficient sampling from the posterior of this mo…
A new method for the computation of the posterior distribution of the number k of components in a finite mixture is presented. Two aspects of prior specification are also studied: an argument is made for the use of a Poisson(1) distribution as the prior for k; and methods are given for the selection of hyperparameter values in the mixture of normals model, with natural conjugate priors on the components parameters.
Institut de Mathématiques de Bordeaux, CNRS, Université de Bordeaux, Bordeaux INP（波尔多数学研究所、法国国家科学研究中心、波尔多大学、波尔多国立理工学院）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Patch priors have become an important component of image restoration. A powerful approach in this category of restoration algorithms is the popular Expected Patch Log-Likelihood (EPLL) algorithm. EPLL uses a Gaussian mixture model (GMM) prior learned on clean image patches as a way to regularize degraded patches. In this paper, we show that a generalized Gaussian mixture model (GGMM) captures the underlying distribution of patches better than a GMM. Even though GGMM is a powerful prior to combine with EPLL, the non-Gaussianity of its components presents major challenges to be applied to a comp…
This paper studies Bayesian variable selection in linear models with general spherically symmetric error distributions. We propose sub-harmonic priors which arise as a class of mixtures of Zellner's g-priors for which the Bayes factors are independent of the underlying error distribution, as long as it is in the spherically symmetric class. Because of this invariance to spherically symmetric error distribution, we refer to our method as a robust Bayesian variable selection method. We demonstrate that our Bayes factors have model selection consistency and are coherent. We also develop Laplace a…
This paper adopts a Bayesian nonparametric mixture model where the mixing distribution belongs to the wide class of normalized homogeneous completely random measures. We propose a truncation method for the mixing distribution by discarding the weights of the unnormalized measure smaller than a threshold. We prove convergence in law of our approximation, provide some theoretical properties and characterize its posterior distribution so that a blocked Gibbs sampler is devised. The versatility of the approximation is illustrated by two different applications. In the first the normalized Bessel ra…
One of the main approaches used to construct prior distributions for objective Bayes methods is the concept of random imaginary observations. Under this setup, the expected-posterior prior (EPP) offers several advantages, among which it has a nice and simple interpretation and provides an effective way to establish compatibility of priors among models. In this paper, we study the power-expected posterior prior as a generalization to the EPP in objective Bayesian model selection under normal linear models. We prove that it can be represented as a mixture of $g$-prior, like a wide range of prior…
When using mixture models it may be the case that the modeller has a-priori beliefs or desires about what the components of the mixture should represent. For example, if a mixture of normal densities is to be fitted to some data, it may be desirable for components to focus on capturing differences in location rather than scale. We introduce a framework called proximity penalty priors (PPPs) that allows this preference to be made explicit in the prior information. The approach is scale-free and imposes minimal restrictions on the posterior; in particular no arbitrary thresholds need to be set. …
Choosing the number of mixture components remains an elusive challenge. Model selection criteria can be either overly liberal or conservative and return poorly-separated components of limited practical use. We formalize non-local priors (NLPs) for mixtures and show how they lead to well-separated components with non-negligible weight, interpretable as distinct subpopulations. We also propose an estimator for posterior model probabilities under local and non-local priors, showing that Bayes factors are ratios of posterior to prior empty-cluster probabilities. The estimator is widely applicable …
We propose a prior distribution for the number of components of a finite mixture model. The novelty is that the prior distribution is obtained by considering the loss one would incur if the true value representing the number of components were not considered. The prior has an elegant and easy to implement structure, which allows to naturally include any prior information one may have as well as to opt for a default solution in cases where this information is not available. The performance of the prior, and comparison with existing alternatives, is studied through the analysis of both real and …
Scientists and engineers are often interested in learning the number of subpopulations (or components) present in a data set. A common suggestion is to use a finite mixture model (FMM) with a prior on the number of components. Past work has shown the resulting FMM component-count posterior is consistent; that is, the posterior concentrates on the true, generating number of components. But consistency requires the assumption that the component likelihoods are perfectly specified, which is unrealistic in practice. In this paper, we add rigor to data-analysis folk wisdom by proving that under eve…
A new approach on the joint estimation of partially exchangeable observations is presented by constructing pairwise dependence between $m$ random density functions, each of which is modeled as a mixture of geometric stick breaking processes. This approach is based on a new random central masses version of the Pairwise Dependent Dirichlet Process prior mixture model (PDDP) first introduced in Hatjispyros et al. (2011). The idea is to create pairwise dependence through random measures that are location-preserving-expectations of Dirichlet random measures. Our contention is that mixture modeling …
We assume that a high-dimensional datum, like an image, is a compositional expression of a set of properties, with a complicated non-linear relationship between the datum and its properties. This paper proposes a factorial mixture prior for capturing latent properties, thereby adding structured compositionality to deep generative models. The prior treats a latent vector as belonging to Cartesian product of subspaces, each of which is quantized separately with a Gaussian mixture model. Some mixture components can be set to represent properties as observed random variables whenever labeled prope…
We propose Dirichlet Process Mixture (DPM) models for prediction and cluster-wise variable selection, based on two choices of shrinkage baseline prior distributions for the linear regression coefficients, namely the Horseshoe prior and Normal-Gamma prior. We show in a simulation study that each of the two proposed DPM models tend to outperform the standard DPM model based on the non-shrinkage normal prior, in terms of predictive, variable selection, and clustering accuracy. This is especially true for the Horseshoe model, and when the number of covariates exceeds the within-cluster sample size…
We propose an objective Bayesian approach to the selection of covariates and their penalised splines transformations in generalised additive models. Specification of a reasonable default prior for the model parameters and combination with a multiplicity-correction prior for the models themselves is crucial for this task. Here we use well-studied and well-behaved continuous mixtures of g-priors as default priors. We introduce the methodology in the normal model and extend it to non-normal exponential families. A simulation study and an application from the literature illustrate the proposed app…
We propose a two-component mixture of a noninformative (diffuse) and an informative prior distribution, weighted through the data in such a way to prefer the first component if a prior-data conflict arises. The data-driven approach for computing the mixture weights makes this class data-dependent. Although rarely used with any theoretical motivation, data-dependent priors are often used for different reasons, and their use has been a lot debated over the last decades. However, our approach is justified in terms of Bayesian inference as an approximation of a hierarchical model and as a conditio…
The large-sample properties of likelihood-based statistical inference under mixture models have received much attention from statisticians. Although the consistency of the nonparametric MLE is regarded as a standard conclusion, many researchers ignore the precise conditions required on the mixture model. An incorrect claim of consistency can lead to false conclusions even if the mixture model under investigation seems well behaved. Under a finite normal mixture model, for instance, the consistency of the plain MLE is often erroneously assumed in spite of recent research breakthroughs. This pap…
A natural Bayesian approach for mixture models with an unknown number of components is to take the usual finite mixture model with Dirichlet weights, and put a prior on the number of components---that is, to use a mixture of finite mixtures (MFM). While inference in MFMs can be done with methods such as reversible jump Markov chain Monte Carlo, it is much more common to use Dirichlet process mixture (DPM) models because of the relative ease and generality with which DPM samplers can be applied. In this paper, we show that, in fact, many of the attractive mathematical properties of DPMs are als…
Mixtures of Zellner's g-priors have been studied extensively in linear models and have been shown to have numerous desirable properties for Bayesian variable selection and model averaging. Several extensions of g-priors to Generalized Linear Models (GLMs) have been proposed in the literature; however, the choice of prior distribution of g and resulting properties for inference have received considerably less attention. In this paper, we unify mixtures of g-priors in GLMs by assigning the truncated Compound Confluent Hypergeometric (tCCH) distribution to 1/(1 + g), which encompasses as special …
Item nonresponse is frequently encountered in practice. Ignoring missing data can lose efficiency and lead to misleading inference. Fractional imputation is a frequentist approach of imputation for handling missing data. However, the parametric fractional imputation of \cite{kim2011parametric} may be subject to bias under model misspecification. In this paper, we propose a novel semiparametric fractional imputation method using Gaussian mixture models. The proposed method is computationally efficient and leads to robust estimation. The proposed method is further extended to incorporate the cat…
In model-based-clustering mixture models are used to group data points into clusters. A useful concept introduced for Gaussian mixtures by Malsiner Walli et al (2016) are sparse finite mixtures, where the prior distribution on the weight distribution of a mixture with $K$ components is chosen in such a way that a priori the number of clusters in the data is random and is allowed to be smaller than $K$ with high probability. The number of cluster is then inferred a posteriori from the data. The present paper makes the following contributions in the context of sparse finite mixture modelling. Fi…
The Dirichlet Process Mixture Model (DPMM) is a Bayesian non-parametric approach widely used for density estimation and clustering. In this manuscript, we study the choice of prior for the variance or precision matrix when Gaussian kernels are adopted. Typically, in the relevant literature, the assessment of mixture models is done by considering observations in a space of only a handful of dimensions. Instead, we are concerned with more realistic problems of higher dimensionality, in a space of up to 20 dimensions. We observe that the choice of prior is increasingly important as the dimensiona…
