In this paper, we propose a generalized scale mixture family of distributions, namely the Power Exponential Scale Mixture (PESM) family, to model the sparsity inducing priors currently in use for sparse signal recovery (SSR). We show that the successful and popular methods such as LASSO, Reweighted $\ell_1$ and Reweighted $\ell_2$ methods can be formulated in an unified manner in a maximum a posteriori (MAP) or Type I Bayesian framework using an appropriate member of the PESM family as the sparsity inducing prior. In addition, exploiting the natural hierarchical framework induced by the PESM family, we utilize these priors in a Type II framework and develop the corresponding EM based estimation algorithms. Some insight into the differences between Type I and Type II methods is provided and of particular interest in the algorithmic development is the Type II variant of the popular and successful reweighted $\ell_1$ method. Extensive empirical results are provided and they show that the Type II methods exhibit better support recovery than the corresponding Type I methods.
This paper uses Gaussian mixture model instead of linear Gaussian model to fit the distribution of every node in Bayesian network. We will explain why and how we use Gaussian mixture models in Bayesian network. Meanwhile we propose a new method, called double iteration algorithm, to optimize the mixture model, the double iteration algorithm combines the expectation maximization algorithm and gradient descent algorithm, and it performs perfectly on the Bayesian network with mixture models. In experiments we test the Gaussian mixture model and the optimization algorithm on different graphs which…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Technical University of Munich（慕尼黑工业大学）
Despite the ubiquity of kernel-based clustering, surprisingly few statistical guarantees exist beyond settings that consider strong structural assumptions on the data generation process. In this work, we take a step towards bridging this gap by studying the statistical performance of kernel-based clustering algorithms under non-parametric mixture models. We provide necessary and sufficient separability conditions under which these algorithms can consistently recover the underlying true clustering. Our analysis provides guarantees for kernel clustering approaches without structural assumptions …
Delft University of Technology（代尔夫特理工大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Vanderbilt University（范德比尔特大学）；Vanderbilt University Medical Center（范德比尔特大学医学中心）
We study the community detection problem on a Gaussian mixture model, in which vertices are divided into $k\geq 2$ distinct communities. The major difference in our model is that the intensities for Gaussian perturbations are different for different entries in the observation matrix, and we do not assume that every community has the same number of vertices. We explicitly find the threshold for the exact recovery of the maximum likelihood estimation. Applications include the community detection on hypergraphs.
University of California, San Diego（加州大学圣地亚哥分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Mixture models are widely used to fit complex and multimodal datasets. In this paper we study mixtures with high dimensional sparse latent parameter vectors and consider the problem of support recovery of those vectors. While parameter learning in mixture models is well-studied, the sparsity constraint remains relatively unexplored. Sparsity of parameter vectors is a natural constraint in variety of settings, and support recovery is a major step towards parameter estimation. We provide efficient algorithms for support recovery that have a logarithmic sample complexity dependence on the dimensi…
University of Glasgow（格拉斯哥大学）；Amazon Research（亚马逊研究院）
We propose a new probabilistic method for unsupervised recovery of corrupted data. Given a large ensemble of degraded samples, our method recovers accurate posteriors of clean values, allowing the exploration of the manifold of possible reconstructed data and hence characterising the underlying uncertainty. In this setting, direct application of classical variational methods often gives rise to collapsed densities that do not adequately explore the solution space. Instead, we derive our novel reduced entropy condition approximate inference method that results in rich posteriors. We test our mo…
We determine the information-theoretic cutoff value on separation of cluster centers for exact recovery of cluster labels in a $K$-component Gaussian mixture model with equal cluster sizes. Moreover, we show that a semidefinite programming (SDP) relaxation of the $K$-means clustering method achieves such sharp threshold for exact recovery without assuming the symmetry of cluster centers.
Spatiotemporal Clustering with Neyman-Scott Processes via Connections to Bayesian Nonparametric Mixture Models
University of Michigan（密歇根大学）；Stanford University（斯坦福大学）；New York University（纽约大学）；Flatiron Institute（弗拉蒂伦研究所）；Wu Tsai Neurosciences Institute（吴蔡神经科学研究所）
Insight Centre for Data Analytics（洞察数据分析中心）；University College Dublin（都柏林大学）；Maynooth University（梅努斯大学）
This paper focuses on the problem of hierarchical non-overlapping clustering of a dataset. In such a clustering, each data item is associated with exactly one leaf node and each internal node is associated with all the data items stored in the sub-tree beneath it, so that each level of the hierarchy corresponds to a partition of the dataset. We develop a novel Bayesian nonparametric method combining the nested Chinese Restaurant Process (nCRP) and the Hierarchical Dirichlet Process (HDP). Compared with other existing Bayesian approaches, our solution tackles data with complex latent mixture fe…
KTH Royal Institute of Technology（瑞典皇家理工学院）；RaySearch Laboratories（瑞镭实验室）；Silo AI（Silo AI（西洛人工智能公司））
We present a new nonparametric mixture-of-experts model for multivariate regression problems, inspired by the probabilistic k-nearest neighbors algorithm. Using a conditionally specified model, predictions for out-of-sample inputs are based on similarities to each observed data point, yielding predictive distributions represented by Gaussian mixtures. Posterior inference is performed on the parameters of the mixture components as well as the distance metric using a mean-field variational Bayes algorithm accompanied with a stochastic gradient-based optimization procedure. The proposed method is…
Variational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models
Center for Data Science Waseda University（早稻田大学数据科学中心）
We present a proposal to deal with the non-normality issue in the context of regression models with measurement errors when both the response and the explanatory variable are observed with error. We extend the normal model by jointly modeling the unobserved covariate and the random errors by a finite mixture of scale mixture of skew-normal distributions. This approach allows us to model data with great flexibility, accommodating skewness, heavy tails, and multi-modality.
We study Bayesian estimation of mixture models and argue in favor of fitting the marginal posterior distribution over component assignments directly, rather than Gibbs sampling from the joint posterior on components and parameters as is commonly done. Some previous authors have found the former approach to have slow mixing, but we show that, implemented correctly, it can achieve excellent performance. In particular, we describe a new Monte Carlo algorithm for sampling from the marginal posterior of a general integrable mixture that makes use of rejection-free sampling from the prior over compo…
For two vast families of mixture distributions and a given prior, we provide unified representations of posterior and predictive distributions. Model applications presented include bivariate mixtures of Gamma distributions labelled as Kibble-type, non-central Chi-square and F distributions, the distribution of $R^2$ in multiple regression, variance mixture of normal distributions, and mixtures of location-scale exponential distributions including the multivariate Lomax distribution. An emphasis is also placed on analytical representations and the relationships with a host of existing distribut…
Revealing underlying relations between nodes in a network is one of the most important tasks in network analysis. Using tools and techniques from a variety of disciplines, many community recovery methods have been developed for different scenarios. Despite the recent interest on community recovery in multilayer networks, theoretical results on the accuracy of the estimates are few and far between. Given a multilayer, e.g. temporal, network and a multilayer stochastic block model, we derive bounds for sufficient separation between intra- and inter-block connectivity parameters to achieve poster…
