Multivariate normal mixtures provide a flexible method of fitting high-dimensional data. It is shown that their topography, in the sense of their key features as a density, can be analyzed rigorously in lower dimensions by use of a ridgeline manifold that contains all critical points, as well as the ridges of the density. A plot of the elevations on the ridgeline shows the key features of the mixed density. In addition, by use of the ridgeline, we uncover a function that determines the number of modes of the mixed density when there are two components being mixed. A followup analysis then gives a curvature function that can be used to prove a set of modality theorems.
In this paper, we propose finite mixtures of multivariate skew Laplace distributions to model both skewness and heavy-tailedness in the heterogeneous data sets. The maximum likelihood estimators for the parameters of interest are obtained by using the EM algorithm. We give a small simulation study and a real data example to illustrate the performance of the proposed mixture model.
Robust estimation under multivariate normal (MVN) mixture model is always a computational challenge. A recently proposed maximum pseudo \b{eta}-likelihood estimator aims to estimate the unknown parameters of a MVN mixture model in the spirit of minimum density power divergence (DPD) methodology but with a relatively simpler and tractable computational algorithm even for larger dimensions. In this letter, we will rigorously derive the existence and weak consistency of the maximum pseudo \b{eta}-likelihood estimator in case of MVN mixture models under a reasonable set of assumptions.
We consider a multivariate finite mixture of Gaussian regression models for high-dimensional data, where the number of covariates and the size of the response may be much larger than the sample size. We provide an $\ell_1$-oracle inequality satisfied by the Lasso estimator according to the Kullback-Leibler loss. This result is an extension of the $\ell_1$-oracle inequality established by Meynet in \cite{Meynet} in the multivariate case. We focus on the Lasso for its $\ell_1$-regularization properties rather than for the variable selection procedure, as it was done in Städler in \cite{Stadler}.
For the nonparametric estimation of multivariate finite mixture models with the conditional independence assumption, we propose a new formulation of the objective function in terms of penalized smoothed Kullback-Leibler distance. The nonlinearly smoothed majorization-minimization (NSMM) algorithm is derived from this perspective. An elegant representation of the NSMM algorithm is obtained using a novel projection-multiplication operator, a more precise monotonicity property of the algorithm is discovered, and the existence of a solution to the main optimization problem is proved for the first …
Shannon and Renyi entropies are quantitative measures of uncertainty in a data set. They are developed by Renyi in the context of entropy theory. These measures have been studied in the case of the multivariate t-distributions. We extend these tools to the class of multivariate skew t-distributions and then to more families of finite mixture of multivariate skew t distributions. In particular, by using generalized Holder inequality and some properties of multinomial theorem, we find upper and lower bounds of entropies for these families. An approximate value of these entropies can be calculate…
We show that rate-adaptive multivariate density estimation can be performed using Bayesian methods based on Dirichlet mixtures of normal kernels with a prior distribution on the kernel's covariance matrix parameter. We derive sufficient conditions on the prior specification that guarantee convergence to a true density at a rate that is optimal minimax for the smoothness class to which the true density belongs. No prior knowledge of smoothness is assumed. The sufficient conditions are shown to hold for the Dirichlet location mixture of normals prior with a Gaussian base measure and an inverse-W…
Density estimation represents one of the most successful applications of Bayesian nonparametrics. In particular, Dirichlet process mixtures of normals are the gold standard for density estimation and their asymptotic properties have been studied extensively, especially in the univariate case. However a gap between practitioners and the current theoretical literature is present. So far, posterior asymptotic results in the multivariate case are available only for location mixtures of Gaussian kernels with independent prior on the common covariance matrix, while in practice as well as from a conc…
It is shown that a simple Dirichlet process mixture of multivariate normals offers Bayesian density estimation with adaptive posterior convergence rates. Toward this, a novel sieve for non-parametric mixture densities is explored, and its rate adaptability to various smoothness classes of densities in arbitrary dimension is demonstrated. This sieve construction is expected to offer a substantial technical advancement in studying Bayesian non-parametric mixture models based on stick-breaking priors.
Two statistical problems for multivariate mixture distributions
Universidad de San Andrés, Argentina and Pedeciba Matemática, Uruguay（圣安德烈斯大学，阿根廷及乌拉圭数学发展委员会）；Instituto de Estadística, Departamento de Métodos Cuantitativos, FCEA, Universidad de la República, Uruguay（乌拉圭共和国大学经济学院定量方法系统计研究所）；Département de mathématiques et de statistique, Université Laval, Québec City, G1V 0A6, Québec, Canada（拉瓦尔大学数学与统计系）
In the problem of selecting variables in a multivariate linear regression model, we derive new Bayesian information criteria based on a prior mixing a smooth distribution and a delta distribution. Each of them can be interpreted as a fusion of the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Inheriting their asymptotic properties, our information criteria are consistent in variable selection in both the large-sample and the high-dimensional asymptotic frameworks. In numerical simulations, variable selection methods based on our information criteria choose th…
This paper addresses the problem of full model estimation for non-parametric finite mixture models. It presents an approach for selecting the number of components and the subset of discriminative variables (i.e., the subset of variables having different distributions among the mixture components). The proposed approach considers a discretization of each variable into B bins and a penalization of the resulting log-likelihood. Considering that the number of bins tends to infinity as the sample size tends to infinity, we prove that our estimator of the model (number of components and subset of re…
An important and yet difficult problem in fitting multivariate mixture models is determining the mixture complexity. We develop theory and a unified framework for finding the nonparametric maximum likelihood estimator of a multivariate mixing distribution and consequently estimating the mixture complexity. Multivariate mixtures provide a flexible approach to fitting high-dimensional data while offering data reduction through the number, location and shape of the component densities. The central principle of our method is to cast the mixture maximization problem in the concave optimization fram…
Finite mixtures of multivariate normal distributions have been widely used in empirical applications in diverse fields such as statistical genetics and statistical finance. Testing the number of components in multivariate normal mixture models is a long-standing challenge even in the most important case of testing homogeneity. This paper develops likelihood-based tests of the null hypothesis of $M_0$ components against the alternative hypothesis of $M_0 + 1$ components for a general $M_0 \geq 1$. For heteroscedastic normal mixtures, we propose an EM test and derive the asymptotic distribution …
In this paper, a new mixture family of multivariate normal distributions, formed by mixing multivariate normal distribution and skewed distribution, is constructed. Some properties of this family, such as characteristic function, moment generating function, and the first four moments are derived. The distributions of affine transformations and canonical forms of the model are also derived. An EM type algorithm is developed for the maximum likelihood estimation of model parameters. We have considered in detail, some special cases of the family, using standard gamma and standard exponential mixt…
We prove minimax bounds for estimating Gaussian location mixtures on $\mathbb{R}^d$ under the squared $L^2$ and the squared Hellinger loss functions. Under the squared $L^2$ loss, we prove that the minimax rate is upper and lower bounded by a constant multiple of $n^{-1}(\log n)^{d/2}$. Under the squared Hellinger loss, we consider two subclasses based on the behavior of the tails of the mixing measure. When the mixing measure has a sub-Gaussian tail, the minimax rate under the squared Hellinger loss is bounded from below by $(\log n)^{d}/n$. On the other hand, when the mixing measure is only …
We give sufficient identifiability conditions for estimating mixing proportions in two-component mixtures of skew normal distributions with one known component. We consider the univariate case as well as two multivariate extensions: a multivariate skew normal distribution (MSN) by Azzalini and Dalla Valle (1996) and the canonical fundamental skew normal distribution (CFUSN) by Arellano-Valle and Genton (2005). The characteristic function of the CFUSN distribution is additionally derived.
Multivariate normal mixtures provide a flexible model for high-dimensional data. They are widely used in statistical genetics, statistical finance, and other disciplines. Due to the unboundedness of the likelihood function, classical likelihood-based methods, which may have nice practical properties, are inconsistent. In this paper, we recommend a penalized likelihood method for estimating the mixing distribution. We show that the maximum penalized likelihood estimator is strongly consistent when the number of components has a known upper bound. We also explore a convenient EM-algorithm for co…
Several formulations have long existed in the literature in the form of continuous mixtures of normal variables where a mixing variable operates on the mean or on the variance or on both the mean and the variance of a multivariate normal variable, by changing the nature of these basic constituents from constants to random quantities. More recently, other mixture-type constructions have been introduced, where the core random component, on which the mixing operation operates, is not necessarily normal. The main aim of the present work is to show that many existing constructions can be encompasse…
In an industrial context, the activity of sensors is recorded at a high frequency. A challenge is to automatically detect abnormal measurement behavior. Considering the sensor measures as functional data, the problem can be formulated as the detection of outliers in a multivariate functional data set. Due to the heterogeneity of this data set, the proposed contaminated mixture model both clusters the multivariate functional data into homogeneous groups and detects outliers. The main advantage of this procedure over its competitors is that it does not require to specify the proportion of outlie…
Suppose that $\m{U} = (U_1, \ldots , U_d) $ has a Uniform$([0,1]^d)$ distribution, that $\m{Y} = (Y_1 , \ldots , Y_d) $ has the distribution $G$ on $\RR_+^d$, and let $\m{X} = (X_1 , \ldots , X_d) = (U_1 Y_1 , \ldots , U_d Y_d )$. The resulting class of distributions of $\m{X}$ (as $G$ varies over all distributions on $\RR_+^d$) is called the {\sl Scale Mixture of Uniforms} class of distributions, and the corresponding class of densities on $\RR_+^d$ is denoted by $\{\cal F}_{SMU}(d)$. We study maximum likelihood estimation in the family ${\cal F}_{SMU}(d)$. We prove existence of the MLE, esta…
We study frequentist risk properties of predictive density estimators for mean mixtures of multivariate normal distributions, involving an unknown location parameter $θ\in \mathbb{R}^d$, and which include multivariate skew normal distributions. We provide explicit representations for Bayesian posterior and predictive densities, including the benchmark minimum risk equivariant (MRE) density, which is minimax and generalized Bayes with respect to an improper uniform density for $θ$. For four dimensions or more, we obtain Bayesian densities that improve uniformly on the MRE density under Kullback…
On Mixtures of Three Homoscedastic Gaussian Densities: An Unconditional Sharper Bound on the Number of Modes
Institute of Statistical Mathematics（统计数学研究所）；The Graduate University for Advanced Studies, SOKENDAI（advanced studies大学，SOKENDAI）；RIKEN（理化学研究所）；Kagoshima University（鹿儿岛大学）
We propose a novel estimator for the number of components (denoted by $M$) in a K-variate non-parametric finite mixture model, where the analyst has repeated observations of $K\geq2$ variables that are independent given a finitely supported unobserved variable. Under a mild assumption on the joint distribution of the observed and latent variables, we show that an integral operator $T$, that is identified from the data, has rank equal to $M$. Using this observation, and the fact that singular values are stable under perturbations, the estimator of $M$ that we propose is based on a thresholding …
A constructive proof of identification of multilinear decompositions of multiway arrays is presented. It can be applied to show identification in a variety of multivariate latent structures. Examples are finite-mixture models and hidden Markov models. The key step to show identification is the joint diagonalization of a set of matrices in the same nonorthogonal basis. An estimator of the latent-structure model may then be based on a sample version of this joint-diagonalization problem. Algorithms are available for computation and we derive distribution theory. We further develop asymptotic the…
This paper studies nonparametric estimation of parameters of multivariate Hawkes processes. We consider the Bayesian setting and derive posterior concentration rates. First rates are derived for L1-metrics for stochastic intensities of the Hawkes process. We then deduce rates for the L1-norm of interactions functions of the process. Our results are exemplified by using priors based on piecewise constant functions, with regular or random partitions and priors based on mixtures of Betas distributions. Numerical illustrations are then proposed with in mind applications for inferring functional co…
Although continuous density estimation has received abundant attention in the Bayesian nonparametrics literature, there is limited theory on multivariate mixed scale density estimation. In this note, we consider a general framework to jointly model continuous, count and categorical variables under a nonparametric prior, which is induced through rounding latent variables having an unknown density with respect to Lebesgue measure. For the proposed class of priors, we provide sufficient conditions for large support, strong consistency and rates of posterior contraction. These conditions allow one…
Parametric distributions are an important part of statistics. There is now a voluminous literature on different fascinating formulations of flexible distributions. We present a selective and brief overview of a small subset of these distributions, focusing on those that are obtained by scaling the mean and/or covariance matrix of the (multivariate) normal distribution with some scaling variable(s). Namely, we consider the families of mean mixture, variance mixture, and mean-variance mixture of normal distributions. Its basic properties, some notable special/limiting cases, and parameter estima…
Brown's 1971 paper "Admissible estimators, recurrent diffusions and insoluble boundary value problems" is a landmark in the admissibility literature. It nearly completely settles the issue of admissibility/inadmissibility for estimating the mean of a multivariate normal distribution with identity covariance under sum of squared error loss. We revisit this wonderful tour de force on its 50th anniversary and present an alternative and more direct proof of the result for generalized Bayes estimators corresponding to priors which are a subclass of scale mixtures of spherical normals.
The multivariate version of the Mixed Tempered Stable is proposed. It is a generalization of the Normal Variance Mean Mixtures. Characteristics of this new distribution and its capacity in fitting tails and capturing dependence structure between components are investigated. We discuss a random number generating procedure and introduce an estimation methodology based on the minimization of a distance between empirical and theoretical characteristic functions. Asymptotic tail behavior of the univariate Mixed Tempered Stable is exploited in the estimation procedure in order to obtain a better mod…
Normal variance-mean mixtures encompass a large family of useful distributions such as the generalized hyperbolic distribution, which itself includes the Student t, Laplace, hyperbolic, normal inverse Gaussian, and variance gamma distributions as special cases. We study shape properties of normal variance-mean mixtures, in both the univariate and multivariate cases, and determine conditions for unimodality and log-concavity of the density functions. This leads to a short proof of the unimodality of all generalized hyperbolic densities. We also interpret such results in practical terms and disc…
