We develop a mixture model for transition density approximation, together with soft model selection, in the presence of noisy and heterogeneous nonlinear dynamics. Our model builds on the Gaussian mixture transition distribution (MTD) model for continuous state spaces, extending component means with nonlinear functions that are modeled using Gaussian process (GP) priors. The resulting model flexibly captures nonlinear and heterogeneous lag dependence when several mixture components are active, identifies low-order nonlinear dependence while inferring relevant lags when few components are active, and averages over multiple and competing single-lag models to quantify/propagate uncertainty. Sparsity-inducing priors on the mixture weights aid in selecting a subset of active lags. The hierarchical model specification follows conventions for both GP regression and MTD models, admitting a convenient Gibbs sampling scheme for posterior inference. We demonstrate properties of the proposed model with two simulated and two real time series, emphasizing approximation of lag-dependent transition densities and model selection. In most cases, the model decisively recovers important features. The proposed model provides a simple, yet flexible framework that preserves useful and distinguishing characteristics of the MTD model class.
Clustering time series into similar groups can improve models by combining information across like time series. While there is a well developed body of literature for clustering of time series, these approaches tend to generate clusters independently of model training which can lead to poor model fit. We propose a novel distributed approach that simultaneously clusters and fits autoregression models for groups of similar individuals. We apply a Wishart mixture model so as to cluster individuals while modeling the corresponding autocovariance matrices at the same time. The fitted Wishart scale …
The standard approach to analyzing brain electrical activity is to examine the spectral density function (SDF) and identify predefined frequency bands that have the most substantial relative contributions to the overall variance of the signal. However, a limitation of this approach is that the precise frequency and bandwidth of oscillations vary with cognitive demands. Thus they should not be arbitrarily defined a priori in an experiment. In this paper, we develop a data-driven approach that identifies (i) the number of prominent peaks, (ii) the frequency peak locations, and (iii) their corres…
In this paper we consider the problem of dynamic clustering, where cluster memberships may change over time and clusters may split and merge over time, thus creating new clusters and destroying existing ones. We propose a Bayesian nonparametric approach to dynamic clustering via mixture modeling. Our approach relies on a novel time-dependent nonparametric prior defined by combining: i) a copula-based transformation of a Gaussian autoregressive process; ii) the stick-breaking construction of the Dirichlet process. Posterior inference is performed through a particle Markov chain Monte Carlo algo…
This paper deals with Bayesian inference of a mixture of Gaussian distributions. A novel formulation of the mixture model is introduced, which includes the prior constraint that each Gaussian component is always assigned a minimal number of data points. This enables noninformative improper priors such as the Jeffreys prior to be placed on the component parameters. We demonstrate difficulties involved in specifying a prior for the standard Gaussian mixture model, and show how the new model can be used to overcome these. MCMC methods are given for efficient sampling from the posterior of this mo…
Expanding a lower-dimensional problem to a higher-dimensional space and then projecting back is often beneficial. This article rigorously investigates this perspective in the context of finite mixture models, namely how to improve inference for mixture models by using auxiliary variables. Despite the large literature in mixture models and several empirical examples, there is no previous work that gives general theoretical justification for including auxiliary variables in mixture models, even for special cases. We provide a theoretical basis for comparing inference for mixture multivariate mod…
Multi-output regression models must exploit dependencies between outputs to maximise predictive performance. The application of Gaussian processes (GPs) to this setting typically yields models that are computationally demanding and have limited representational power. We present the Gaussian Process Autoregressive Regression (GPAR) model, a scalable multi-output GP model that is able to capture nonlinear, possibly input-varying, dependencies between outputs in a simple and tractable way: the product rule is used to decompose the joint distribution over the outputs into a set of conditionals, e…
In the mixture modeling frame, this paper presents the polynomial Gaussian cluster-weighted model (CWM). It extends the linear Gaussian CWM, for bivariate data, in a twofold way. Firstly, it allows for possible nonlinear dependencies in the mixture components by considering a polynomial regression. Secondly, it is not restricted to be used for model-based clustering only being contextualized in the most general model-based classification framework. Maximum likelihood parameter estimates are derived using the EM algorithm and model selection is carried out using the Bayesian information criteri…
This paper introduces a fully Bayesian analysis of mixture autoregressive models with Student t components. With the capacity of capturing the behaviour in the tails of the distribution, the Student t MAR model provides a more flexible modelling framework than its Gaussian counterpart, leading to fitted models with fewer parameters and of easier interpretation. The degrees of freedom are also treated as random variables, and hence are included in the estimation process.
This paper is concerned with an important issue in finite mixture modelling, the selection of the number of mixing components. We propose a new penalized likelihood method for model selection of finite multivariate Gaussian mixture models. The proposed method is shown to be statistically consistent in determining of the number of components. A modified EM algorithm is developed to simultaneously select the number of components and to estimate the mixing weights, i.e. the mixing probabilities, and unknown parameters of Gaussian distributions. Simulations and a real data analysis are presented t…
Obtaining reliable estimates of conditional covariance matrices is an important task of heteroskedastic multivariate time series. In portfolio optimization and financial risk management, it is crucial to provide measures of uncertainty and risk as accurately as possible. We propose using mixture vector autoregressive (MVAR) models for portfolio optimization. Combining a mixture of distributions that depend on the recent history of the process, MVAR models can accommodate asymmetry, multimodality, heteroskedasticity and cross-correlation in multivariate time series data. For mixtures of Normal …
This paper uses Gaussian mixture model instead of linear Gaussian model to fit the distribution of every node in Bayesian network. We will explain why and how we use Gaussian mixture models in Bayesian network. Meanwhile we propose a new method, called double iteration algorithm, to optimize the mixture model, the double iteration algorithm combines the expectation maximization algorithm and gradient descent algorithm, and it performs perfectly on the Bayesian network with mixture models. In experiments we test the Gaussian mixture model and the optimization algorithm on different graphs which…
Heavy-tailed distributions are widely used in robust mixture modelling due to possessing thick tails. As a computationally tractable subclass of the stable distributions, sub-Gaussian $α$-stable distribution received much interest in the literature. Here, we introduce a type of expectation maximization algorithm that estimates parameters of a mixture of sub-Gaussian stable distributions. A comparative study, in the presence of some well-known mixture models, is performed to show the robustness and performance of the mixture of sub-Gaussian $α$-stable distributions for modelling, simulated, syn…
Deep learning is a hierarchical inference method formed by subsequent multiple layers of learning able to more efficiently describe complex relationships. In this work, Deep Gaussian Mixture Models are introduced and discussed. A Deep Gaussian Mixture model (DGMM) is a network of multiple layers of latent variables, where, at each layer, the variables follow a mixture of Gaussian distributions. Thus, the deep mixture model consists of a set of nested mixtures of linear models, which globally provide a nonlinear model able to describe the data in a very flexible way. In order to avoid overparam…
Vector-valued Gaussian mixtures form an important special subset of vector-valued distributions. In general, vector-valued distributions constitute natural representations for physical entities, which can mutate or transit among alternative manifestations distributed in a given space. A key example is color imagery. In this note, we vectorize the Gaussian mixture model and study several different optimal mass transport related problems associated to such models. The benefits of using vector Gaussian mixture for optimal mass transport include computational efficiency and the ability to preserve…
Non-Gaussian mixture models are gaining increasing attention for mixture model-based clustering particularly when dealing with data that exhibit features such as skewness and heavy tails. Here, such a mixture distribution is presented, based on the multivariate normal inverse Gaussian (MNIG) distribution. For parameter estimation of the mixture, a Bayesian approach via Gibbs sampler is used; for this, a novel approach to simulate univariate generalized inverse Gaussian random variables and matrix generalized inverse Gaussian random matrices is provided. The proposed algorithm will be applied t…
We quantify the parameter stability of a spherical Gaussian Mixture Model (sGMM) under small perturbations in distribution space. Namely, we derive the first explicit bound to show that for a mixture of spherical Gaussian $P$ (sGMM) in a pre-defined model class, all other sGMM close to $P$ in this model class in total variation distance has a small parameter distance to $P$. Further, this upper bound only depends on $P$. The motivation for this work lies in providing guarantees for fitting Gaussian mixtures; with this aim in mind, all the constants involved are well defined and distribution fr…
The mixture of Gaussian distributions, a soft version of k-means , is considered a state-of-the-art clustering algorithm. It is widely used in computer vision for selecting classes, e.g., color, texture, and shapes. In this algorithm, each class is described by a Gaussian distribution, defined by its mean and covariance. The data is described by a weighted sum of these Gaussian distributions. We propose a new method, inspired by quantum interference in physics. Instead of modeling each class distribution directly, we model a class wave function such that its magnitude square is the class Gauss…
Gaussian mixture models are central to classical statistics, widely used in the information sciences, and have a rich mathematical structure. We examine their maximum likelihood estimates through the lens of algebraic statistics. The MLE is not an algebraic function of the data, so there is no notion of ML degree for these models. The critical points of the likelihood function are transcendental, and there is no bound on their number, even for mixtures of two univariate Gaussians.
We determine the information-theoretic cutoff value on separation of cluster centers for exact recovery of cluster labels in a $K$-component Gaussian mixture model with equal cluster sizes. Moreover, we show that a semidefinite programming (SDP) relaxation of the $K$-means clustering method achieves such sharp threshold for exact recovery without assuming the symmetry of cluster centers.
Gaussian mixture models are widely used in Statistics. A fundamental aspect of these distributions is the study of the local maxima of the density, or modes. In particular, it is not known how many modes a mixture of $k$ Gaussians in $d$ dimensions can have. We give a brief account of this problem's history. Then, we give improved lower bounds and the first upper bound on the maximum number of modes, provided it is finite.
Gaussian mixture models find their place as a powerful tool, mostly in the clustering problem, but with proper preparation also in feature extraction, pattern recognition, image segmentation and in general machine learning. When faced with the problem of schema matching, different mixture models computed on different pieces of data can maintain crucial information about the structure of the dataset. In order to measure or compare results from mixture models, the Wasserstein distance can be very useful, however it is not easy to calculate for mixture distributions. In this paper we derive one o…
KTH Royal Institute of Technology（KTH皇家理工学院）；Schlumberger Cambridge Research（斯伦贝谢剑桥研究中心）
We propose a greedy variational method for decomposing a non-negative multivariate signal as a weighted sum of Gaussians, which, borrowing the terminology from statistics, we refer to as a Gaussian mixture model. Notably, our method has the following features: (1) It accepts multivariate signals, i.e. sampled multivariate functions, histograms, time series, images, etc. as input. (2) The method can handle general (i.e. ellipsoidal) Gaussians. (3) No prior assumption on the number of mixture components is needed. To the best of our knowledge, no previous method for Gaussian mixture model decomp…
This paper presents a new model called infinite mixtures of multivariate Gaussian processes, which can be used to learn vector-valued functions and applied to multitask learning. As an extension of the single multivariate Gaussian process, the mixture model has the advantages of modeling multimodal data and alleviating the computationally cubic complexity of the multivariate Gaussian process. A Dirichlet process prior is adopted to allow the (possibly infinite) number of mixture components to be automatically inferred from training data, and Markov chain Monte Carlo sampling techniques are use…
This work provides a computationally efficient and statistically consistent moment-based estimator for mixtures of spherical Gaussians. Under the condition that component means are in general position, a simple spectral decomposition technique yields consistent parameter estimates from low-order observable moments, without additional minimum separation assumptions needed by previous computationally efficient estimation procedures. Thus computational and information-theoretic barriers to efficient estimation in mixture models are precluded when the mixture components have means in general posit…
We address the problem to estimate a Kronecker graphical model corresponding to an autoregressive Gaussian stochastic process. The latter is completely described by the power spectral density function whose inverse has support which admits a Kronecker product decomposition. We propose a Bayesian approach to estimate such a model. We test the effectiveness of the proposed method by some numerical experiments. We also apply the procedure to urban pollution monitoring data.
We study the community detection problem on a Gaussian mixture model, in which vertices are divided into $k\geq 2$ distinct communities. The major difference in our model is that the intensities for Gaussian perturbations are different for different entries in the observation matrix, and we do not assume that every community has the same number of vertices. We explicitly find the threshold for the exact recovery of the maximum likelihood estimation. Applications include the community detection on hypergraphs.
Sony Computer Science Laboratories, Inc.（索尼计算机科学实验室有限公司）；Tokyo（东京）
The traditional Minkowski distances are induced by the corresponding Minkowski norms in real-valued vector spaces. In this work, we propose novel statistical symmetric distances based on the Minkowski's inequality for probability densities belonging to Lebesgue spaces. These statistical Minkowski distances admit closed-form formula for Gaussian mixture models when parameterized by integer exponents. This result extends to arbitrary mixtures of exponential families with natural parameter spaces being cones: This includes the binomial, the multinomial, the zero-centered Laplacian, the Gaussian a…
Mixture - modeling of mass spectra is an approach with many potential applications including peak detection and quantification, smoothing, de-noising, feature extraction and spectral signal compression. However, existing algorithms do not allow for automatic analyses of whole spectra. Therefore, despite highlighting potential advantages of mixture modeling of mass spectra of peptide/protein mixtures and some preliminary results presented in several papers, the mixture modeling approach was so far not developed to the stage enabling systematic comparisons with existing software packages for pro…
How can we train a statistical mixture model on a massive data set? In this work we show how to construct coresets for mixtures of Gaussians. A coreset is a weighted subset of the data, which guarantees that models fitting the coreset also provide a good fit for the original data set. We show that, perhaps surprisingly, Gaussian mixtures admit coresets of size polynomial in dimension and the number of mixture components, while being independent of the data set size. Hence, one can harness computationally intensive algorithms to compute a good approximation on a significantly smaller data set. …
We consider the problem of estimating means of two Gaussians in a 2-Gaussian mixture, which is not balanced and is corrupted by noise of an arbitrary distribution. We present a robust algorithm to estimate the parameters, together with upper bounds on the numbers of samples required for the estimate to be correct, where the bounds are parametrised by the dimension, ratio of the mixing coefficients, a measure of the separation of the two Gaussians, related to Mahalanobis distance, and a condition number of the covariance matrix. In theory, this is the first sample-complexity result for imbalanc…
We study the convergence behavior of the Expectation Maximization (EM) algorithm on Gaussian mixture models with an arbitrary number of mixture components and mixing weights. We show that as long as the means of the components are separated by at least $Ω(\sqrt{\min\{M,d\}})$, where $M$ is the number of components and $d$ is the dimension, the EM algorithm converges locally to the global optimum of the log-likelihood. Further, we show that the convergence rate is linear and characterize the size of the basin of attraction to the global optimum.
Setting initial values of parameters of mixture distributions estimated by using the EM recursive algorithm is very important to the overall quality of estimation. None of the existing methods is suitable for mixtures with large number of components. We present a relevant methodology of estimating initial values of parameters of univariate, heteroscedastic Gaussian mixtures, on the basis of the dynamic programming algorithm for partitioning the range of observations into bins. We evaluate variants of dynamic programming method corresponding to different scoring functions for partitioning. For …
In this article, normal inverse Gaussian (NIG) autoregressive model is introduced. The parameters of the model are estimated using Expectation Maximization (EM) algorithm. The efficacy of the EM algorithm is shown using simulated and real world financial data. It is shown that NIG autoregressive model fit very well the considered financial data and hence could be useful in modeling of various real life time-series data.
When performing a time series analysis of continuous data, for example from climate or environmental problems, the assumption that the process is Gaussian is often violated. Therefore, we introduce two non-Gaussian autoregressive time series models that are able to fit skewed and heavy-tailed time series data. Our two models are based on the Tukey g-and-h transformation. We discuss parameter estimation, order selection, and forecasting procedures for our models and examine their performances in a simulation study. We demonstrate the usefulness of our models by applying them to two sets of wind…
Lung tumor tracking for radiotherapy requires real-time, multiple-step ahead forecasting of a quasi-periodic time series recording instantaneous tumor locations. We introduce a location-mixture autoregressive (LMAR) process that admits multimodal conditional distributions, fast approximate inference using the EM algorithm and accurate multiple-step ahead predictive distributions. LMAR outperforms several commonly used methods in terms of out-of-sample prediction accuracy using clinical data from lung tumor patients. With its superior predictive performance and real-time computation, the LMAR m…
Testing for regime switching when the regime switching probabilities are specified either as constants (`mixture models') or are governed by a finite-state Markov chain (`Markov switching models') are long-standing problems that have also attracted recent interest. This paper considers testing for regime switching when the regime switching probabilities are time-varying and depend on observed data (`observation-dependent regime switching'). Specifically, we consider the likelihood ratio test for observation-dependent regime switching in mixture autoregressive models. The testing problem is hig…
A new mixture autoregressive model based on Student's $t$-distribution is proposed. A key feature of our model is that the conditional $t$-distributions of the component models are based on autoregressions that have multivariate $t$-distributions as their (low-dimensional) stationary distributions. That autoregressions with such stationary distributions exist is not immediate. Our formulation implies that the conditional mean of each component model is a linear function of past observations and the conditional variance is also time varying. Compared to previous mixture autoregressive models ou…
Mixture autoregressive (MAR) models provide a flexible way to model time series with predictive distributions which depend on the recent history of the process and are able to accommodate asymmetry and multimodality. Bayesian inference for such models offers the additional advantage of incorporating the uncertainty in the estimated models into the predictions. We introduce a new way of sampling from the posterior distribution of the parameters of MAR models which allows for covering the complete parameter space of the models, unlike previous approaches. We also propose a relabelling algorithm …
In this note we discuss (Gaussian) intrinsic conditional autoregressive (CAR) models for disconnected graphs, with the aim of providing practical guidelines for how these models should be defined, scaled and implemented. We show how these suggestions can be implemented in two examples on disease mapping.
