The class of location-scale finite mixtures is of enduring interest both from applied and theoretical perspectives of probability and statistics. We prove the following results: to an arbitrary degree of accuracy, (a) location-scale mixtures of a continuous probability density function (PDF) can approximate any continuous PDF, uniformly, on a compact set; and (b) for any finite $p\ge1$, location-scale mixtures of an essentially bounded PDF can approximate any PDF in $\mathcal{L}_{p}$, in the $\mathcal{L}_{p}$ norm.
Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.
Nowadays in density estimation, posterior rates of convergence for location and location-scale mixtures of Gaussians are only known under light-tail assumptions; with better rates achieved by location mixtures. It is conjectured, but not proved, that the situation should be reversed under heavy tails assumptions. The conjecture is based on the feeling that there is no need to achieve a good order of approximation in regions with few data (say, in the tails), favoring location-scale mixtures which allow for spatially varying order of approximation. Here we test the previous argument on the Gaus…
We present general results on the univariate tail conditional expectation (TCE) and multivariate tail conditional expectation for location-scale mixture of elliptical distributions. Examples include the location-scale mixture of normal distributions, location-scale mixture of Student-$t$ distributions, location-scale mixture of Logistic distributions and location-scale mixture of Laplace distributions. We also consider portfolio risk decomposition with TCE for location-scale mixture of elliptical distributions.
Density estimation represents one of the most successful applications of Bayesian nonparametrics. In particular, Dirichlet process mixtures of normals are the gold standard for density estimation and their asymptotic properties have been studied extensively, especially in the univariate case. However a gap between practitioners and the current theoretical literature is present. So far, posterior asymptotic results in the multivariate case are available only for location mixtures of Gaussian kernels with independent prior on the common covariance matrix, while in practice as well as from a conc…
We study location-scale mixture priors for nonparametric statistical problems, including multivariate regression, density estimation and classification. We show that a rate-adaptive procedure can be obtained if the prior is properly constructed. In particular, we show that adaptation is achieved if a kernel mixture prior on a regression function is constructed using a Gaussian kernel, an inverse gamma bandwidth, and Gaussian mixing weights.
In finite mixtures of location-scale distributions, if there is no constraint on the parameters then the maximum likelihood estimate does not exist. But when the ratios of the scale parameters are restricted appropriately, the maximum likelihood estimate exists. We prove that the maximum likelihood estimator (MLE) is strongly consistent, if the ratios of the scale parameters are restricted from below by $\exp(-n^{d}), 0 < d < 1 $, where $n$ is the sample size.
In a finite mixture of location-scale distributions maximum likelihood estimator does not exist because of the unboundedness of the likelihood function when the scale parameter of some mixture component approaches zero. In order to study the strong consistency of maximum likelihood estimator, we consider the case that the scale parameters of the component distributions are restricted from below by $c_n$, where $c_n$ is a sequence of positive real numbers which tend to zero as the sample size $n$ increases. We prove that under mild regularity conditions maximum likelihood estimator is strongly …
We prove minimax bounds for estimating Gaussian location mixtures on $\mathbb{R}^d$ under the squared $L^2$ and the squared Hellinger loss functions. Under the squared $L^2$ loss, we prove that the minimax rate is upper and lower bounded by a constant multiple of $n^{-1}(\log n)^{d/2}$. Under the squared Hellinger loss, we consider two subclasses based on the behavior of the tails of the mixing measure. When the mixing measure has a sub-Gaussian tail, the minimax rate under the squared Hellinger loss is bounded from below by $(\log n)^{d}/n$. On the other hand, when the mixing measure is only …
ML-estimation based on mixtures of Normal distributions is a widely used tool for cluster analysis. However, a single outlier can make the parameter estimation of at least one of the mixture components break down. Among others, the estimation of mixtures of t-distributions by McLachlan and Peel [Finite Mixture Models (2000) Wiley, New York] and the addition of a further mixture component accounting for ``noise'' by Fraley and Raftery [The Computer J. 41 (1998) 578-588] were suggested as more robust alternatives. In this paper, the definition of an adequate robustness measure for cluster analys…
In finite mixtures of location-scale distributions, if there is no constraint or penalty on the parameters, then the maximum likelihood estimator does not exist because the likelihood is unbounded. To avoid this problem, we consider a penalized likelihood, where the penalty is a function of the minimum of the ratios of the scale parameters and the sample size. It is shown that the penalized maximum likelihood estimator is strongly consistent. We also analyze the consistency of a penalized maximum likelihood estimator where the penalty is imposed on the scale parameters themselves.
This article discusses the problem of estimation of parameters in finite mixtures when the mixture components are assumed to be symmetric and to come from the same location family. We refer to these mixtures as semi-parametric because no additional assumptions other than symmetry are made regarding the parametric form of the component distributions. Because the class of symmetric distributions is so broad, identifiability of parameters is a major issue in these mixtures. We develop a notion of identifiability of finite mixture models, which we call k-identifiability, where k denotes the number…
We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in $d$-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accuracy of every NPMLE. Our results imply, in particular, that every NPMLE achieves near parametric risk (up to logarithmic multiplicative factors) when the true density is a discrete Gaussian mixture without any prior information on the number of mixture components. NPMLEs can natural…
This paper proposes a unified class of generalized location-scale mixture of multivariate elliptical distributions and studies integral stochastic orderings of random vectors following such distributions. Given a random vector $\boldsymbol{Z}$, independent of $\boldsymbol{X}$ and $\boldsymbol{Y}$, the scale parameter of this class of distributions is mixed with a function $α(\boldsymbol{Z})$ and its skew parameter is mixed with another function $β(\boldsymbol{Z})$. Sufficient (and necessary) conditions are established for stochastically comparing different random vectors stemming from this cla…
We study nonparametric Bayesian inference with location mixtures of the Laplace density and a Dirichlet process prior on the mixing distribution. We derive a contraction rate of the corresponding posterior distribution, both for the mixing distribution relative to the Wasserstein metric and for the mixed density relative to the Hellinger and $L_q$ metrics.
In this article, we revisit the problem of estimating the unknown zero-symmetric distribution in a two-component location mixture model, considered in previous works, now under the assumption that the zero-symmetric distribution has a log-concave density. When consistent estimators for the shift locations and mixing probability are used, we show that the nonparametric log-concave Maximum Likelihood estimator (MLE) of both the mixed density and that of the unknown zero-symmetric component are consistent in the Hellinger distance. In case the estimators for the shift locations and mixing probabi…
The Expectation-Maximization (EM) algorithm has been predominantly used to approximate the maximum likelihood estimation of the location-scale Gaussian mixtures. However, when the models are over-specified, namely, the chosen number of components to fit the data is larger than the unknown true number of components, EM needs a polynomial number of iterations in terms of the sample size to reach the final statistical radius; this is computationally expensive in practice. The slow convergence of EM is due to the missing of the locally strong convexity with respect to the location parameter on the…
We derive uniform convergence rates for the maximum likelihood estimator and minimax lower bounds for parameter estimation in two-component location-scale Gaussian mixture models with unequal variances. We assume the mixing proportions of the mixture are known and fixed, but make no separation assumption on the underlying mixture components. A phase transition is shown to exist in the optimal parameter estimation rate, depending on whether or not the mixture is balanced. Key to our analysis is a careful study of the dependence between the parameters of location-scale Gaussian mixture models, a…
We consider the problem of Bayesian density estimation on the positive semiline for possibly unbounded densities. We propose a hierarchical Bayesian estimator based on the gamma mixture prior which can be viewed as a location mixture. We study convergence rates of Bayesian density estimators based on such mixtures. We construct approximations of the local Hölder densities, and of their extension to unbounded densities, to be continuous mixtures of gamma distributions, leading to approximations of such densities by finite mixtures. These results are then used to derive posterior concentration r…
This paper studies the optimal rate of estimation in a finite Gaussian location mixture model in high dimensions without separation conditions. We assume that the number of components $k$ is bounded and that the centers lie in a ball of bounded radius, while allowing the dimension $d$ to be as large as the sample size $n$. Extending the one-dimensional result of Heinrich and Kahn \cite{HK2015}, we show that the minimax rate of estimating the mixing distribution in Wasserstein distance is $Θ((d/n)^{1/4} + n^{-1/(4k-2)})$, achieved by an estimator computable in time $O(nd^2+n^{5/4})$. Furthermor…
This paper studies identifiability and convergence behaviors for parameters of multiple types in finite mixtures, and the effects of model fitting with extra mixing components. First, we present a general theory for strong identifiability, which extends from the previous work of Nguyen [2013] and Chen [1995] to address a broad range of mixture models and to handle matrix-variate parameters. These models are shown to share the same Wasserstein distance based optimal rates of convergence for the space of mixing distributions --- $n^{-1/2}$ under $W_1$ for the exact-fitted and $n^{-1/4}$ under $W…
Flexible spatial models that allow transitions between tail dependence classes have recently appeared in the literature. However, inference for these models is computationally prohibitive, even in moderate dimensions, due to the necessity of repeatedly evaluating the multivariate Gaussian distribution function. In this work, we attempt to achieve truly high-dimensional inference for extremes of spatial processes, while retaining the desirable flexibility in the tail dependence structure, by modifying an established class of models based on scale mixtures Gaussian processes. We show that the de…
We study frequentist risk properties of predictive density estimators for mean mixtures of multivariate normal distributions, involving an unknown location parameter $θ\in \mathbb{R}^d$, and which include multivariate skew normal distributions. We provide explicit representations for Bayesian posterior and predictive densities, including the benchmark minimum risk equivariant (MRE) density, which is minimax and generalized Bayes with respect to an improper uniform density for $θ$. For four dimensions or more, we obtain Bayesian densities that improve uniformly on the MRE density under Kullback…
An often-cited fact regarding mixing or mixture distributions is that their density functions are able to approximate the density function of any unknown distribution to arbitrary degrees of accuracy, provided that the mixing or mixture distribution is sufficiently complex. This fact is often not made concrete. We investigate and review theorems that provide approximation bounds for mixing distributions. Connections between the approximation bounds of mixing distributions and estimation bounds for the maximum likelihood estimator of finite mixtures of location- scale distributions are reviewed…
We show that rate-adaptive multivariate density estimation can be performed using Bayesian methods based on Dirichlet mixtures of normal kernels with a prior distribution on the kernel's covariance matrix parameter. We derive sufficient conditions on the prior specification that guarantee convergence to a true density at a rate that is optimal minimax for the smoothness class to which the true density belongs. No prior knowledge of smoothness is assumed. The sufficient conditions are shown to hold for the Dirichlet location mixture of normals prior with a Gaussian base measure and an inverse-W…
Although continuous density estimation has received abundant attention in the Bayesian nonparametrics literature, there is limited theory on multivariate mixed scale density estimation. In this note, we consider a general framework to jointly model continuous, count and categorical variables under a nonparametric prior, which is induced through rounding latent variables having an unknown density with respect to Lebesgue measure. For the proposed class of priors, we provide sufficient conditions for large support, strong consistency and rates of posterior contraction. These conditions allow one…
This work studies the location estimation problem for a mixture of two rotation invariant log-concave densities. We demonstrate that Least Squares EM, a variant of the EM algorithm, converges to the true location parameter from a randomly initialized point. We establish the explicit convergence rates and sample complexity bounds, revealing their dependence on the signal-to-noise ratio and the tail property of the log-concave distribution. Moreover, we show that this global convergence property is robust under model mis-specification. Our analysis generalizes previous techniques for proving the…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Columbia University（哥伦比亚大学）
We consider the problem of estimating the common mean of independently sampled data, where samples are drawn in a possibly non-identical manner from symmetric, unimodal distributions with a common mean. This generalizes the setting of Gaussian mixture modeling, since the number of distinct mixture components may diverge with the number of observations. We propose an estimator that adapts to the level of heterogeneity in the data, achieving near-optimality in both the i.i.d. setting and some heterogeneous settings, where the fraction of ``low-noise'' points is as small as $\frac{\log n}{n}$. Ou…
