Estimation of structure, such as in variable selection, graphical modelling or cluster analysis is notoriously difficult, especially for high-dimensional data. We introduce stability selection. It is based on subsampling in combination with (high-dimensional) selection algorithms. As such, the method is extremely general and has a very wide range of applicability. Stability selection provides finite sample control for some error rates of false discoveries and hence a transparent principle to choose a proper amount of regularisation for structure estimation. Variable selection and structure estimation improve markedly for a range of selection methods if stability selection is applied. We prove for randomised Lasso that stability selection will be variable selection consistent even if the necessary conditions needed for consistency of the original Lasso method are violated. We demonstrate stability selection for variable selection and Gaussian graphical modelling, using real and simulated data.
This paper proposes a penalized composite likelihood method for model selection in colored graphical Gaussian models. The method provides a sparse and symmetry-constrained estimator of the precision matrix, and thus conducts model selection and precision matrix estimation simultaneously. In particular, the method uses penalty terms to constrain the elements of the precision matrix, which enables us to transform the model selection problem into a constrained optimization problem. Further, computer experiments are conducted to illustrate the performance of the proposed new methodology. It is sho…
In recent years, Ising prior with the network information for the "in" or "out" binary random variable in Bayesian variable selections has received more and more attentions. In this paper, we discover that even without the informative prior a Bayesian variable selection problem itself can be considered as a complete graph and described by a Ising model with random interactions. There are many advantages of treating variable selection as a graphical model, such as it is easy to employ the single site updating as well as the cluster updating algorithm, suitable for problems with small sample siz…
The mplot package provides an easy to use implementation of model stability and variable inclusion plots (Müller and Welsh 2010; Murray, Heritier, and Müller 2013) as well as the adaptive fence (Jiang, Rao, Gu, and Nguyen 2008; Jiang, Nguyen, and Rao 2009) for linear and generalised linear models. We provide a number of innovations on the standard procedures and address many practical implementation issues including the addition of redundant variables, interactive visualisations and approximating logistic models with linear models. An option is provided that combines our bootstrap approach wit…
Recently, a special case of precision matrix estimation based on a distributionally robust optimization (DRO) framework has been shown to be equivalent to the graphical lasso. From this formulation, a method for choosing the regularization term, i.e., for graphical model selection, was proposed. In this work, we establish a theoretical connection between the confidence level of graphical model selection via the DRO formulation and the asymptotic family-wise error rate of estimating false edges. Simulation experiments and real data analyses illustrate the utility of the asymptotic family-wise e…
This paper introduces an estimator of the relative directed distance between an estimated model and the true model, based on the Kulback-Leibler divergence and is motivated by the generalized information criterion proposed by Konishi and Kitagawa. This estimator can be used to select model in penalized Gaussian copula graphical models. The use of this estimator is not feasible for high-dimensional cases. However, we derive an efficient way to compute this estimator which is feasible for the latter class of problems. Moreover, this estimator is, generally, appropriate for several penalties such…
Rice University（莱斯大学）；Baylor College of Medicine（贝勒医学院）
In neuroscience, researchers seek to uncover the connectivity of neurons from large-scale neural recordings or imaging; often people employ graphical model selection and estimation techniques for this purpose. But, existing technologies can only record from a small subset of neurons leading to a challenging problem of graph selection in the presence of extensive latent variables. Chandrasekaran et al. (2012) proposed a convex program to address this problem that poses challenges from both a computational and statistical perspective. To solve this problem, we propose an incredibly simple soluti…
Selecting regularization parameters in penalized high-dimensional graphical models in a principled, data-driven, and computationally efficient manner continues to be one of the key challenges in high-dimensional statistics. We present substantial computational gains and conceptual generalizations of the Stability Approach to Regularization Selection (StARS), a state-of-the-art graphical model selection scheme. Using properties of the Poisson-Binomial distribution and convex non-asymptotic distributional modeling we propose lower and upper bounds on the StARS graph regularization path which res…
In this paper we penetrate and extend the notion of local constancy in graphical models that has been introduced by Honorio et al. (2009). We propose Neighborhood-Fused Lasso, a method for model selection in high-dimensional graphical models, leveraging locality information. Our approach is based on an extension of the idea of node-wise regression (Meinshausen-Bühlmann, 2006) by adding a fusion penalty. We propose a fast numerical algorithm for our approach, and provide theoretical and numerical evidence for the fact that our methodology outperforms related approaches that are ignoring the loc…
This paper considers a high-dimensional linear regression problem where there are complex correlation structures among predictors. We propose a graph-constrained regularization procedure, named Sparse Laplacian Shrinkage with the Graphical Lasso Estimator (SLS-GLE). The procedure uses the estimated precision matrix to describe the specific information on the conditional dependence pattern among predictors, and encourages both sparsity on the regression model and the graphical model. We introduce the Laplacian quadratic penalty adopting the graph information, and give detailed discussions on th…
In a Gaussian graphical model, the conditional independence between two variables are characterized by the corresponding zero entries in the inverse covariance matrix. Maximum likelihood method using the smoothly clipped absolute deviation (SCAD) penalty (Fan and Li, 2001) and the adaptive LASSO penalty (Zou, 2006) have been proposed in literature. In this article, we establish the result that using Bayesian information criterion (BIC) to select the tuning parameter in penalized likelihood estimation with both types of penalties can lead to consistent graphical model selection. We compare the …
Università Cattolica del Sacro Cuore（圣心天主教大学）；Università degli Studi di Torino（都灵大学）
This paper proposes a new algorithm for an automatic variable selection procedure in High Dimensional Graphical Models. The algorithm selects the relevant variables for the node of interest on the basis of mutual information. Several contributions in literature have investigated the use of mutual information in selecting the appropriate number of relevant features in a large data-set, but most of them have focused on binary outcomes or required high computational effort. The algorithm here proposed overcomes these drawbacks as it is an extension of Chow and Liu's algorithm. Once, the probabili…
We consider the problem of covariance matrix estimation in the presence of latent variables. Under suitable conditions, it is possible to learn the marginal covariance matrix of the observed variables via a tractable convex program, where the concentration matrix of the observed variables is decomposed into a sparse matrix (representing the graphical structure of the observed variables) and a low rank matrix (representing the marginalization effect of latent variables). We present an efficient first-order method based on split Bregman to solve the convex problem. The algorithm is guaranteed to…
KU Leuven（鲁汶大学）；Leuven Statistics Research Center (LStat)（鲁汶统计研究中心）；Université Paris Descartes（巴黎笛卡尔大学）；CNRS（法国国家科学研究中心）；Université Paris-Sud（巴黎南大学）；INRA（法国国家农业研究院）
Gaussian graphical models are widely utilized to infer and visualize networks of dependencies between continuous variables. However, inferring the graph is difficult when the sample size is small compared to the number of variables. To reduce the number of parameters to estimate in the model, we propose a non-asymptotic model selection procedure supported by strong theoretical guarantees based on an oracle inequality and a minimax lower bound. The covariance matrix of the model is approximated by a block-diagonal matrix. The structure of this matrix is detected by thresholding the sample covar…
Chandrasekaran, Parrilo and Willsky (2010) proposed a convex optimization problem to characterize graphical model selection in the presence of unobserved variables. This convex optimization problem aims to estimate an inverse covariance matrix that can be decomposed into a sparse matrix minus a low-rank matrix from sample data. Solving this convex optimization problem is very challenging, especially for large problems. In this paper, we propose two alternating direction methods for solving this problem. The first method is to apply the classical alternating direction method of multipliers to s…
A challenging problem in estimating high-dimensional graphical models is to choose the regularization parameter in a data-dependent way. The standard techniques include $K$-fold cross-validation ($K$-CV), Akaike information criterion (AIC), and Bayesian information criterion (BIC). Though these methods work well for low-dimensional problems, they are not suitable in high dimensional settings. In this paper, we present StARS: a new stability-based method for choosing the regularization parameter in high dimensional inference for undirected graphs. The method has a clear interpretation: we use t…
The topic of this paper is modeling and analyzing dependence in stochastic social networks. Using a latent variable block model allows the analysis of dependence between blocks via the analysis of a latent graphical model. Our approach to the analysis of the graphical model then is based on the idea underlying the neighborhood selection scheme put forward by Meinshausen and Bühlmann (2006). However, because of the latent nature of our model, estimates have to be used in lieu of the unobserved variables. This leads to a novel analysis of graphical models under uncertainty, in the spirit of Rose…
This thesis studies two problems in modern statistics. First, we study selective inference, or inference for hypothesis that are chosen after looking at the data. The motiving application is inference for regression coefficients selected by the lasso. We present the Condition-on-Selection method that allows for valid selective inference, and study its application to the lasso, and several other selection algorithms. In the second part, we consider the problem of learning the structure of a pairwise graphical model over continuous and discrete variables. We present a new pairwise model for grap…
Though Gaussian graphical models have been widely used in many scientific fields, relatively limited progress has been made to link graph structures to external covariates. We propose a Gaussian graphical regression model, which regresses both the mean and the precision matrix of a Gaussian graphical model on covariates. In the context of co-expression quantitative trait locus (QTL) studies, our method can determine how genetic variants and clinical conditions modulate the subject-level network structures, and recover both the population-level and subject-level gene networks. Our framework enc…
We consider the task of estimating a Gaussian graphical model in the high-dimensional setting. The graphical lasso, which involves maximizing the Gaussian log likelihood subject to an l1 penalty, is a well-studied approach for this task. We begin by introducing a surprising connection between the graphical lasso and hierarchical clustering: the graphical lasso in effect performs a two-step procedure, in which (1) single linkage hierarchical clustering is performed on the variables in order to identify connected components, and then (2) an l1-penalized log likelihood is maximized on the subset …
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Building on a recent framework for distributionally robust optimization, we consider estimation of the inverse covariance matrix for multivariate data. We provide a novel notion of a Wasserstein ambiguity set specifically tailored to this estimation problem, leading to a tractable class of regularized estimators. Special cases include penalized likelihood estimators for Gaussian data, specifically the graphical lasso estimator. As a consequence of this formulation, the radius of the Wasserstein ambiguity set is directly related to the regularization parameter in the estimation problem. Using t…
We show that the two-stage adaptive Lasso procedure (Zou, 2006) is consistent for high-dimensional model selection in linear and Gaussian graphical models. Our conditions for consistency cover more general situations than those accomplished in previous work: we prove that restricted eigenvalue conditions (Bickel et al., 2008) are also sufficient for sparse structure estimation.
Model selection and learning the structure of graphical models from the data sample constitutes an important field of probabilistic graphical model research, as in most of the situations the structure is unknown and has to be learnt from the given dataset. In this paper, we present a new forward model selection algorithm for graphical log-linear models. We use mutual conditional independence check to reduce the search space which also takes care of the evaluation of the joint effects and chances of missing important interactions are eliminated. We illustrate our algorithm with a real dataset e…
Lawrence Berkeley National Lab（劳伦斯伯克利国家实验室）；Stanford University（斯坦福大学）；University of Florida（佛罗里达大学）
Sparse high dimensional graphical model selection is a popular topic in contemporary machine learning. To this end, various useful approaches have been proposed in the context of $\ell_1$-penalized estimation in the Gaussian framework. Though many of these inverse covariance estimation approaches are demonstrably scalable and have leveraged recent advances in convex optimization, they still depend on the Gaussian functional form. To address this gap, a convex pseudo-likelihood based partial correlation graph estimation method (CONCORD) has been recently proposed. This method uses coordinate-wi…
Recently, many regularized procedures have been proposed for variable selection in linear regression, but their performance depends on the tuning parameter selection. Here a criterion for the tuning parameter selection is proposed, which combines the strength of both stability selection and cross-validation and therefore is referred as the prediction and stability selection (PASS). The selection consistency is established assuming the data generating model is a subset of the full model, and the small sample performance is demonstrated through some simulation studies where the assumption is eit…
We introduce Graphical TREX (GTREX), a novel method for graph estimation in high-dimensional Gaussian graphical models. By conducting neighborhood selection with TREX, GTREX avoids tuning parameters and is adaptive to the graph topology. We compare GTREX with standard methods on a new simulation set-up that is designed to assess accurately the strengths and shortcomings of different methods. These simulations show that a neighborhood selection scheme based on Lasso and an optimal (in practice unknown) tuning parameter outperforms other standard methods over a large spectrum of scenarios. Moreo…
Graphical models are used in a variety of problems to uncover hidden structures. There is a huge number of different identification procedures, constructed for different purposes. However, it is important to research different properties of such procedures and compare them in order to find out the best procedure or the best use case for some specific procedure. In this paper, some statistical identification procedures are compared using different measures, such as Type I and Type II errors, ROC AUC.
Covariance graphical lasso applies a lasso penalty on the elements of the covariance matrix. This method is useful because it not only produces sparse estimation of covariance matrix but also discovers marginal independence structures by generating zeros in the covariance matrix. We propose and explore two new algorithms for solving the covariance graphical lasso problem. Our new algorithms are based on coordinate descent and ECM. We show that these two algorithms are more attractive than the only existing competing algorithm of Bien and Tibshirani (2011) in terms of simplicity, speed and stab…
We present an objective Bayes method for covariance selection in Gaussian multivariate regression models whose error term has a covariance structure which is Markov with respect to a Directed Acyclic Graph (DAG). The scope is covariate-adjusted sparse graphical model selection, a topic of growing importance especially in the area of genetical genomics (eQTL analysis). Specifically, we provide a closed-form expression for the marginal likelihood of any DAG (with small parent sets) whose computation virtually requires no subjective elicitation by the user and involves only conjugate matrix norma…
We propose Bayesian methods for Gaussian graphical models that lead to sparse and adaptively shrunk estimators of the precision (inverse covariance) matrix. Our methods are based on lasso-type regularization priors leading to parsimonious parameterization of the precision matrix, which is essential in several applications involving learning relationships among the variables. In this context, we introduce a novel type of selection prior that develops a sparse structure on the precision matrix by making most of the elements exactly zero, in addition to ensuring positive definiteness -- thus cond…
We consider the problem of estimating the parameters in a pairwise graphical model in which the distribution of each node, conditioned on the others, may have a different parametric form. In particular, we assume that each node's conditional distribution is in the exponential family. We identify restrictions on the parameter space required for the existence of a well-defined joint density, and establish the consistency of the neighbourhood selection approach for graph reconstruction in high dimensions when the true underlying graph is sparse. Motivated by our theoretical results, we investigat…
We consider a class of colored graphical Gaussian models obtained by placing symmetry constraints on the precision matrix in a Bayesian framework. The prior distribution on the precision matrix is the colored $G$-Wishart prior which is the Diaconis-Ylvisaker conjugate prior. In this paper, we develop a computationally efficient model search algorithm which combines linear regression with a double reversible jump Markov chain Monte Carlo (MCMC) method. The latter is to estimate the Bayes factors expressed as the ratio of posterior probabilities of two competing models. We also establish the asy…
We propose an AdaPtive Noise Augmentation (PANDA) technique to regularize the estimation and construction of undirected graphical models. PANDA iteratively optimizes the objective function given the noise augmented data until convergence to achieve regularization on model parameters. The augmented noises can be designed to achieve various regularization effects on graph estimation, such as the bridge (including lasso and ridge), elastic net, adaptive lasso, and SCAD penalization; it also realizes the group lasso and fused ridge. We examine the tail bound of the noise-augmented loss function an…
This paper presents a new algorithm for automatic variables selection. In particular, using the Graphical Models properties it is possible to develop a method that can be used in the contest of large dataset. The advantage of this algorithm is that can be combined with different forecasting models. In this research we have used the OLS method and we have compared the result with the LASSO method.
In many applications concerning statistical graphical models the data originate from several subpopulations that share similarities but have also significant differences. This raises the question of how to estimate several graphical models simultaneously. Compiling all the data together to estimate a single graph would ignore the differences among subpopulations. On the other hand, estimating a graph from each subpopulation separately does not make efficient use of the common structure in the data. We develop a new method for simultaneous estimation of multiple graphical models by estimating t…
In variable or graph selection problems, finding a right-sized model or controlling the number of false positives is notoriously difficult. Recently, a meta-algorithm called Stability Selection was proposed that can provide reliable finite-sample control of the number of false positives. Its benefits were demonstrated when used in conjunction with the lasso and orthogonal matching pursuit algorithms. In this paper, we investigate the applicability of stability selection to structured selection algorithms: the group lasso and the structured input-output lasso. We find that using stability selec…
As evidenced by various recent and significant papers within the frequentist literature, along with numerous applications in macroeconomics, genomics, and neuroscience, there continues to be substantial interest to understand the theoretical estimation properties of high-dimensional vector autoregression (VAR) models. To date, however, while Bayesian VAR (BVAR) models have been developed and studied empirically (primarily in the econometrics literature) there exist very few theoretical investigations of the repeated sampling properties for BVAR models in the literature. In this direction, we c…
Learning the structure of dependence relations between variables is a pervasive issue in the statistical literature. A directed acyclic graph (DAG) can represent a set of conditional independences, but different DAGs may encode the same set of relations and are indistinguishable using observational data. Equivalent DAGs can be collected into classes, each represented by a partially directed graph known as essential graph (EG). Structure learning directly conducted on the EG space, rather than on the allied space of DAGs, leads to theoretical and computational benefits. Still, the majority of e…
There has been an intense development of Bayes graphical model estimation approaches over the past decade - however, most of the existing methods are restricted to moderate dimensions. We propose a novel approach suitable for high dimensional settings, by decoupling model fitting and covariance selection. First, a full model based on a complete graph is fit under novel class of continuous shrinkage priors on the precision matrix elements, which induces shrinkage under an equivalence with Cholesky-based regularization while enabling conjugate updates of entire precision matrices. Subsequently, …
Gaussian graphical models are used for determining conditional relationships between variables. This is accomplished by identifying off-diagonal elements in the inverse-covariance matrix that are non-zero. When the ratio of variables (p) to observations (n) approaches one, the maximum likelihood estimator of the covariance matrix becomes unstable and requires shrinkage estimation. Whereas several classical (frequentist) methods have been introduced to address this issue, fully Bayesian methods remain relatively uncommon in practice and methodological literatures. Here we introduce a Bayesian m…
Methods for selecting loglinear models were among Steve Fienberg's research interests since the start of his long and fruitful career. After we dwell upon the string of papers focusing on loglinear models that can be partly attributed to Steve's contributions and influential ideas, we develop a new algorithm for selecting graphical loglinear models that is suitable for analyzing hyper-sparse contingency tables. We show how multi-way contingency tables can be used to represent patterns of human mobility. We analyze a dataset of geolocated tweets from South Africa that comprises 46 million latit…
Multivariate density estimation and graphical models play important roles in statistical learning. The estimated density can be used to construct a graphical model that reveals conditional relationships whereas a graphical structure can be used to build models for density estimation. Our goal is to construct a consolidated framework that can perform both density and graph estimation. Denote $\bm{Z}$ as the random vector of interest with density function $f(\bz)$. Splitting $\bm{Z}$ into two parts, $\bm{Z}=(\bm{X}^T,\bm{Y}^T)^T$ and writing $f(\bz)=f(\bx)f(\by|\bx)$ where $f(\bx)$ is the densit…
Sparse Gaussian graphical models characterize sparse dependence relationships between random variables in a network. To estimate multiple related Gaussian graphical models on the same set of variables, we formulate a hierarchical model, which leads to an optimization problem with a nonconvex log-shift penalty function. We show that under mild conditions the optimization problem is convex despite the inclusion of a nonconvex penalty, and derive an efficient optimization algorithm. Experiments on both synthetic and real data show that the proposed method is able to achieve good selection and est…
Sparse graphical modelling has attained widespread attention across various academic fields. We propose two new graphical model approaches, Gslope and Tslope, which provide sparse estimates of the precision matrix by penalizing its sorted L1-norm, and relying on Gaussian and T-student data, respectively. We provide the selections of the tuning parameters which provably control the probability of including false edges between the disjoint graph components and empirically control the False Discovery Rate for the block diagonal covariance matrices. In extensive simulation and real world analysis,…
A Bayesian approach is used to estimate the covariance matrix of Gaussian data. Ideas from Gaussian graphical models and model selection are used to construct a prior for the covariance matrix that is a mixture over all decomposable graphs. For this prior the probability of each graph size is specified by the user and graphs of equal size are assigned equal probability. Most previous approaches assume that all graphs are equally probable. We show empirically that the prior that assigns equal probability over graph sizes outperforms the prior that assigns equal probability over all graphs, both…
