This paper deals with statistical tests on the components of mixture densities. We propose to test whether the densities of two independent samples of independent random variables $Y_1, ..., Y_n$ and $Z_1, ..., Z_n$ result from the same mixture of $M$ components or not. We provide a test procedure which is proved to be asymptotically optimal according to the minimax setting. We extensively discuss the connection between the mixing weights and the performance of the testing procedure and illustrate it with numerical examples.
We give sufficient identifiability conditions for estimating mixing proportions in two-component mixtures of skew normal distributions with one known component. We consider the univariate case as well as two multivariate extensions: a multivariate skew normal distribution (MSN) by Azzalini and Dalla Valle (1996) and the canonical fundamental skew normal distribution (CFUSN) by Arellano-Valle and Genton (2005). The characteristic function of the CFUSN distribution is additionally derived.
For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of components occurring so far --- that is, the posterior on the number of clusters in the observed data. However, it turns out that this posterior is not consistent --- it does not converge to the true number of components. In this note, we give an elementary demonstration of this inconsist…
Let $(X\_1,\ldots,X\_n)$ be a $d$-dimensional i.i.d sample from a distribution with density $f$. The problem of detection of a two-component mixture is considered. Our aim is to decide whether $f$ is the density of a standard Gaussian random $d$-vector ($f=ϕ\_d$) against $f$ is a two-component mixture: $f=(1-\varepsilon)ϕ\_d +\varepsilon ϕ\_d (.-μ)$ where $(\varepsilon,μ)$ are unknown parameters. Optimal separation conditions on $\varepsilon, μ, n$ and the dimension $d$ are established, allowing to separate both hypotheses with prescribed errors. Several testing procedures are proposed and two…
The posterior distribution of the number of components k in a finite mixture satisfies a set of inequality constraints. The result holds irrespective of the parametric form of the mixture components and under assumptions on the prior distribution weaker than those routinely made in the literature on Bayesian analysis of finite mixtures. The inequality constraints can be used to perform an ``internal'' consistency check of MCMC estimates of the posterior distribution of k and to provide improved estimates which are required to satisfy the constraints. Bounds on the posterior probability of k co…
This article establishes general conditions for posterior consistency of Bayesian finite mixture models with a prior on the number of components. That is, we provide sufficient conditions under which the posterior concentrates on neighborhoods of the true parameter values when the data are generated from a finite mixture over the assumed family of component distributions. Specifically, we establish almost sure consistency for the number of components, the mixture weights, and the component parameters, up to a permutation of the component labels. The approach taken here is based on Doob's theor…
A two-class mixture model, where the density of one of the components is known, is considered. We address the issue of the nonparametric adaptive estimation of the unknown probability density of the second component. We propose a randomly weighted kernel estimator with a fully data-driven bandwidth selection method, in the spirit of the Goldenshluger and Lepski method. An oracle-type inequality for the pointwise quadratic risk is derived as well as convergence rates over Holder smoothness classes. The theoretical results are illustrated by numerical simulations.
BIC criterion is widely used by the neural-network community for model selection tasks, although its convergence properties are not always theoretically established. In this paper we will focus on estimating the number of components in a mixture of multilayer perceptrons and proving the convergence of the BIC criterion in this frame. The penalized marginal-likelihood for mixture models and hidden Markov models introduced by Keribin (2000) and, respectively, Gassiat (2002) is extended to mixtures of multilayer perceptrons for which a penalized-likelihood criterion is proposed. We prove its conv…
This article discusses the problem of estimation of parameters in finite mixtures when the mixture components are assumed to be symmetric and to come from the same location family. We refer to these mixtures as semi-parametric because no additional assumptions other than symmetry are made regarding the parametric form of the component distributions. Because the class of symmetric distributions is so broad, identifiability of parameters is a major issue in these mixtures. We develop a notion of identifiability of finite mixture models, which we call k-identifiability, where k denotes the number…
We study Bayesian estimation of finite mixture models in a general setup where the number of components is unknown and allowed to grow with the sample size. An assumption on growing number of components is a natural one as the degree of heterogeneity present in the sample can grow and new components can arise as sample size increases, allowing full flexibility in modeling the complexity of data. This however will lead to a high-dimensional model which poses great challenges for estimation. We novelly employ the idea of a sample size dependent prior in a Bayesian model and establish a number of…
We consider the problem of detecting sparse heterogeneous mixtures from a nonparametric perspective, and develop distribution-free tests when all effects have the same sign. Specifically, we assume that the null distribution is symmetric about zero, while the true effects have positive median. We evaluate the precise performance of classical tests for the median (t-test, sign test) and classical tests for symmetry (signed-rank, Smirnov, total number of runs, longest run tests) showing that none of them is asymptotically optimal for the normal mixture model in all sparsity regimes. We then sugg…
The problem of estimating the ratio of the means of a two-component Poisson mixture model is considered, when each component is subject to zero-inflation, i.e., excess zero counts. The. resulting {\it zero-inflated Poisson mixture (ZIPM) model} can be treated as a three-component Poisson mixture model with one degenerate component. The EM algorithm is applied to obtain frequentist estimators and their standard errors, the latter determined via an explicit expression for the observed information matrix. Bayes and empirical Bayes estimators also are obtained by means of conjugate priors and thei…
At least seven modes in a heteroscedastic three-component bivariate Gaussian mixture
Graduate School of Science and Engineering, Kagoshima University（鹿儿岛大学理工学研究科）；School of Information and Data Sciences, Nagasaki University（长崎大学信息数据科学学院）；Institute of Statistical Mathematics（统计数学研究所）；The Graduate University for Advanced Studies, SOKENDAI（综合研究大学院大学）；RIKEN（理化学研究所）
We consider a semiparametric mixture of two univariate density functions where one of them is known while the weight and the other function are unknown. Such mixtures have a history of application to the problem of detecting differentially expressed genes under two or more conditions in microarray data. Until now, some additional knowledge about the unknown component (e.g. the fact that it belongs to a location family) has been assumed. As opposed to this approach, we do not assume any additional structure on the unknown density function. For this mixture model, we derive a new sufficient iden…
We study the convergence behavior of the Expectation Maximization (EM) algorithm on Gaussian mixture models with an arbitrary number of mixture components and mixing weights. We show that as long as the means of the components are separated by at least $Ω(\sqrt{\min\{M,d\}})$, where $M$ is the number of components and $d$ is the dimension, the EM algorithm converges locally to the global optimum of the log-likelihood. Further, we show that the convergence rate is linear and characterize the size of the basin of attraction to the global optimum.
We prove that under some regularity and strong identifiability conditions, around a mixing distribution with $m_0$ components, the optimal local minimax rate of estimation of a mixture with $m$ components is $n^{-1/(4(m-m_0) + 2)}$. This corrects a previous paper by Chen (1995) in The Annals of Statistics.
The problem of identifiability of finite mixtures of finite product measures is studied. A mixture model with $K$ mixture components and $L$ observed variables is considered, where each variable takes its value in a finite set with cardinality $M$.The variables are independent in each mixture component. The identifiability of a mixture model means the possibility of attaining the mixture components parameters by observing its mixture distribution. In this paper, we investigate fundamental relations between the identifiability of mixture models and the separability of their observed variables b…
In this paper, we consider a parametric density contamination model. We work with a sample of i.i.d. data with a common density, $f^\star =(1-λ^\star) ϕ+ λ^\star ϕ(.-μ^\star)$, where the shape $ϕ$ is assumed to be known. We establish the optimal rates of convergence for the estimation of the mixture parameters $(λ^\star,μ^\star)$. In particular, we prove that the classical parametric rate $1/\sqrt{n}$ cannot be reached when at least one of these parameters is allowed to tend to $0$ with $n$.
This work is concerned with the detection of a mixture distribution from a $\mathbb{R}$-valued sample. Given a sample $X_1,\dots,X_n$ and an even density $ϕ$, our aim is to detect whether the sample distribution is $ϕ(\cdot-μ)$ for some unknown mean $μ$, or is defined as a two-component mixture based on translations of $ϕ$. We propose a procedure which is based on several spacings of the order statistics, which provides a level-$α$ test for all $n$. Our test is therefore a multiple testing procedure and we prove from a theoretical and practical point of view that it automatically adapts to the…
We derive uniform convergence rates for the maximum likelihood estimator and minimax lower bounds for parameter estimation in two-component location-scale Gaussian mixture models with unequal variances. We assume the mixing proportions of the mixture are known and fixed, but make no separation assumption on the underlying mixture components. A phase transition is shown to exist in the optimal parameter estimation rate, depending on whether or not the mixture is balanced. Key to our analysis is a careful study of the dependence between the parameters of location-scale Gaussian mixture models, a…
In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well-suited for Bayesian density estimation, many investigators are using them for inferences about the number of components, by considering the posterior on the number of components represented in the observed data. We show that this posterior is not consistent --- that is, on data …
When estimating finite mixture models, it is common to make assumptions on the mixture components, such as parametric assumptions. In this work, we make no distributional assumptions on the mixture components and instead assume that observations from the mixture model are grouped, such that observations in the same group are known to be drawn from the same mixture component. We precisely characterize the number of observations $n$ per group needed for the mixture model to be identifiable, as a function of the number $m$ of mixture components. In addition to our assumption-free analysis, we als…
The ability to quantify distinctness of a cluster structure is fundamental for certain simulation studies, in particular for those comparing performance of different classification algorithms. The intrinsic integral measure based on the overlap of corresponding mixture components is often analytically intractable. This is also the case for Gaussian mixture models with unequal covariance matrices when space dimension $d > 1$. In this work we focus on Gaussian mixture models and at the sample level we assume the class assignments to be known. We derive a measure of component overlap based on eig…
Technische Universität Berlin（柏林工业大学）；Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
Recent work has shown that finite mixture models with $m$ components are identifiable, while making no assumptions on the mixture components, so long as one has access to groups of samples of size $2m-1$ which are known to come from the same mixture component. In this work we generalize that result and show that, if every subset of $k$ mixture components of a mixture model are linearly independent, then that mixture model is identifiable with only $(2m-1)/(k-1)$ samples per group. We further show that this value cannot be improved. We prove an analogous result for a stronger form of identifiab…
In this paper, we study convergence properties of the gradient Expectation-Maximization algorithm \cite{lange1995gradient} for Gaussian Mixture Models for general number of clusters and mixing coefficients. We derive the convergence rate depending on the mixing coefficients, minimum and maximum pairwise distances between the true centers and dimensionality and number of components; and obtain a near-optimal local contraction radius. While there have been some recent notable works that derive local convergence rates for EM in the two equal mixture symmetric GMM, in the more general case, the de…
Finite mixtures of multivariate normal distributions have been widely used in empirical applications in diverse fields such as statistical genetics and statistical finance. Testing the number of components in multivariate normal mixture models is a long-standing challenge even in the most important case of testing homogeneity. This paper develops likelihood-based tests of the null hypothesis of $M_0$ components against the alternative hypothesis of $M_0 + 1$ components for a general $M_0 \geq 1$. For heteroscedastic normal mixtures, we propose an EM test and derive the asymptotic distribution …
Estimation of the number of components (or order) of a finite mixture model is a long standing and challenging problem in statistics. We propose the Group-Sort-Fuse (GSF) procedure -- a new penalized likelihood approach for simultaneous estimation of the order and mixing measure in multidimensional finite mixture models. Unlike methods which fit and compare mixtures with varying orders using criteria involving model complexity, our approach directly penalizes a continuous function of the model parameters. More specifically, given a conservative upper bound on the order, the GSF groups and sort…
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；Noah’s Ark Lab（诺亚方舟实验室）；Huawei（华为）
The expectation-maximization (EM) algorithm has been widely used in minimizing the negative log likelihood (also known as cross entropy) of mixture models. However, little is understood about the goodness of the fixed points it converges to. In this paper, we study the regions where one component is missing in two-component mixture models, which we call one-cluster regions. We analyze the propensity of such regions to trap EM and gradient descent (GD) for mixtures of two Gaussians and mixtures of two Bernoullis. In the case of Gaussian mixtures, EM escapes one-cluster regions exponentially fas…
We focus on one-sided, mixture-based stopping rules for the problem of sequential testing a simple null hypothesis against a composite alternative. For the latter, we consider two cases---either a discrete alternative or a continuous alternative that can be embedded into an exponential family. For each case, we find a mixture-based stopping rule that is nearly minimax in the sense of minimizing the maximal Kullback-Leibler information. The proof of this result is based on finding an almost Bayes rule for an appropriate sequential decision problem and on high-order asymptotic approximations for…
Max-mixture processes are defined as Z = max(aX, (1 -- a)Y) with X an asymptotic dependent (AD) process, Y an asymptotic independent (AI) process and a $\in$ [0, 1]. So that, the mixing coefficient a may reveal the strength of the AD part present in the max-mixture process. In this paper we focus on two tests based on censored pairwise likelihood estimates. We compare their performance through an extensive simulation study. Monte Carlo simulation plays a fundamental tool for asymptotic variance calculations. We apply our tests to daily precipitations from the East of Australia. Drawbacks and p…
University of Michigan（密歇根大学）；Technische Universität Berlin（柏林工业大学）
Recent research has established sufficient conditions for finite mixture models to be identifiable from grouped observations. These conditions allow the mixture components to be nonparametric and have substantial (or even total) overlap. This work proposes an algorithm that consistently estimates any identifiable mixture model from grouped observations. Our analysis leverages an oracle inequality for weighted kernel density estimators of the distribution on groups, together with a general result showing that consistent estimation of the distribution on groups implies consistent estimation of m…
Finite mixture models are statistical models which appear in many problems in statistics and machine learning. In such models it is assumed that data are drawn from random probability measures, called mixture components, which are themselves drawn from a probability measure P over probability measures. When estimating mixture models, it is common to make assumptions on the mixture components, such as parametric assumptions. In this paper, we make no assumption on the mixture components, and instead assume that observations from the mixture model are grouped, such that observations in the same …
The test of homogeneity for normal mixtures has been conducted in diverse research areas, but constructing a theory of the test of homogeneity is challenging because the parameter set for the null hypothesis corresponds to singular points in the parameter space. In this paper, we examine this problem from a new perspective and offer a theory of hypothesis testing for homogeneity based on a variational Bayes framework. In the conventional theory, the constant order term of the free energy has remained unknown, however, we clarify its asymptotic behavior because it is necessary for constructing …
In this article, we revisit the problem of estimating the unknown zero-symmetric distribution in a two-component location mixture model, considered in previous works, now under the assumption that the zero-symmetric distribution has a log-concave density. When consistent estimators for the shift locations and mixing probability are used, we show that the nonparametric log-concave Maximum Likelihood estimator (MLE) of both the mixed density and that of the unknown zero-symmetric component are consistent in the Hellinger distance. In case the estimators for the shift locations and mixing probabi…
This paper is concerned with an important issue in finite mixture modelling, the selection of the number of mixing components. We propose a new penalized likelihood method for model selection of finite multivariate Gaussian mixture models. The proposed method is shown to be statistically consistent in determining of the number of components. A modified EM algorithm is developed to simultaneously select the number of components and to estimate the mixing weights, i.e. the mixing probabilities, and unknown parameters of Gaussian distributions. Simulations and a real data analysis are presented t…
Testing for regime switching when the regime switching probabilities are specified either as constants (`mixture models') or are governed by a finite-state Markov chain (`Markov switching models') are long-standing problems that have also attracted recent interest. This paper considers testing for regime switching when the regime switching probabilities are time-varying and depend on observed data (`observation-dependent regime switching'). Specifically, we consider the likelihood ratio test for observation-dependent regime switching in mixture autoregressive models. The testing problem is hig…
Scientists and engineers are often interested in learning the number of subpopulations (or components) present in a data set. A common suggestion is to use a finite mixture model (FMM) with a prior on the number of components. Past work has shown the resulting FMM component-count posterior is consistent; that is, the posterior concentrates on the true, generating number of components. But consistency requires the assumption that the component likelihoods are perfectly specified, which is unrealistic in practice. In this paper, we add rigor to data-analysis folk wisdom by proving that under eve…
This paper focuses on variational inference with intractable likelihood functions that can be unbiasedly estimated. A flexible variational approximation based on Gaussian mixtures is developed, by adopting the mixture population Monte Carlo (MPMC) algorithm in \cite{cappe2008adaptive}. MPMC updates iteratively the parameters of mixture distributions with importance sampling computations, instead of the complicated gradient estimation of the optimization objective in usual variational Bayes. Noticing that MPMC uses a fixed number of mixture components, which is difficult to predict for real app…
The efficacy of family-based approaches to mixture model-based clustering and classification depends on the selection of parsimonious models. Current wisdom suggests the Bayesian information criterion (BIC) for mixture model selection. However, the BIC has well-known limitations, including a tendency to overestimate the number of components as well as a proclivity for, often drastically, underestimating the number of components in higher dimensions. While the former problem might be soluble through merging components, the latter is impossible to mitigate in clustering and classification applic…
