Traditional variable selection methods could fail to be sign consistent when irrepresentable conditions are violated. This is especially critical in high-dimensional settings when the number of predictors exceeds the sample size. In this paper, we propose a new semi-standard partial covariance (SPAC) approach which is capable of reducing correlation effects from other covariates while fully capturing the magnitude of coefficients. The proposed SPAC is effective in choosing covariates which have direct effects on the response variable, while eliminating the predictors which are not directly associated with the response but are highly correlated with the relevant predictors. We show that the proposed SPAC method with the Lasso penalty or the smoothly clipped absolute deviation (SCAD) penalty possesses strong sign consistency in high-dimensional settings. Numerical studies and a post-traumatic stress disorder data application also confirm that the proposed method outperforms the existing Lasso, adaptive Lasso, SCAD, Peter-Clark-simple algorithm, and factor-adjusted regularized model selection methods when the irrepresentable conditions fail.
It is known that for a certain class of single index models (SIMs) $Y = f(\boldsymbol{X}_{p \times 1}^\intercal\boldsymbolβ_0, \varepsilon)$, support recovery is impossible when $\boldsymbol{X} \sim \mathcal{N}(0, \mathbb{I}_{p \times p})$ and a model complexity adjusted sample size is below a critical threshold. Recently, optimal algorithms based on Sliced Inverse Regression (SIR) were suggested. These algorithms work provably under the assumption that the design $\boldsymbol{X}$ comes from an i.i.d. Gaussian distribution. In the present paper we analyze algorithms based on covariance screeni…
We propose a Multi-step Screening Procedure (MSP) for the recovery of sparse linear models in high-dimensional data. This method is based on a repeated small penalty strategy that quickly converges to an estimate within a few iterations. Specifically, in each iteration, an adaptive lasso regression with a small penalty is fit within the reduced feature space obtained from the previous step, rendering its computational complexity roughly comparable with the Lasso. MSP is shown to select the true model under complex correlation structures among the predictors and response, even when the irrepres…
The problem of covariance estimation for replicated surface-valued processes is examined from the functional data analysis perspective. Considerations of statistical and computational efficiency often compel the use of separability of the covariance, even though the assumption may fail in practice. We consider a setting where the covariance structure may fail to be separable locally -- either due to noise contamination or due to the presence of a~non-separable short-range dependent signal component. That is, the covariance is an additive perturbation of a separable component by a~non-separable…
Some Two-Step Procedures for Variable Selection in High-Dimensional Linear Regression
我们研究了高维变量选择问题，通过一些两个步骤的程序。首先，我们表明，给定一些好的初始估计，这是$\ell_{\infty}$ -一致的，但不一定变量选择一致的，我们可以应用非负Garrote，自适应Lasso或硬阈值的过程，以获得一个最终的估计，估计和变量选择一致。与Lasso不同，我们的结果不需要不可表示的条件，即使对于中度$p_n$也可能容易失败（Zhao和Yu，2007），并且它还允许$p_n$几乎与$\exp(n)$一样快地生长（对于硬阈值，对$p_n$没有限制）。我们还研究了岭回归可以用作初始估计的条件。我们证明了在放松的可识别条件下，岭估计是$\ell_{\infty}$ -相容的.当使用$p_n\le n$时，通常满足这样的条件，并且不需要相关和不相关协变量之间的部分正交性，而这是单变量回归所需的（Huang et al.，2008年）。我们的数值研究表明，当使用Lasso或Ridge作为初始估计量时，两步程序具有比使用单变量回归的Lasso或自适应Lasso更高的稀疏恢复率（Huang et al.，2008年）。
High-dimensional compositional data arise naturally in many applications such as metagenomic data analysis. The observed data lie in a high-dimensional simplex, and conventional statistical methods often fail to produce sensible results due to the unit-sum constraint. In this article, we address the problem of covariance estimation for high-dimensional compositional data, and introduce a composition-adjusted thresholding (COAT) method under the assumption that the basis covariance matrix is sparse. Our method is based on a decomposition relating the compositional covariance to the basis covari…
Kronecker PCA involves the use of a space vs. time Kronecker product decomposition to estimate spatio-temporal covariances. In this work the addition of a sparse correction factor is considered, which corresponds to a model of the covariance as a sum of Kronecker products of low (separation) rank and a sparse matrix. This sparse correction extends the diagonally corrected Kronecker PCA of [Greenewald et al 2013, 2014] to allow for sparse unstructured "outliers" anywhere in the covariance matrix, e.g. arising from variables or correlations that do not fit the Kronecker model well, or from sourc…
In sparse principal component analysis we are given noisy observations of a low-rank matrix of dimension $n\times p$ and seek to reconstruct it under additional sparsity assumptions. In particular, we assume here each of the principal components $\mathbf{v}_1,\dots,\mathbf{v}_r$ has at most $s_0$ non-zero entries. We are particularly interested in the high dimensional regime wherein $p$ is comparable to, or even much larger than $n$. In an influential paper, \cite{johnstone2004sparse} introduced a simple algorithm that estimates the support of the principal vectors $\mathbf{v}_1,\dots,\mathbf{…
Bayesian DAG Structure Learning with Simultaneous Shrinkage Covariance Estimation under Scale-Mixture Error Distributions in the Proportional High-Dimensional Regime
Ferdowsi University of Mashhad（马什哈德 Ferdowsi 大学）；North Carolina Agricultural and Technical State University（北卡罗来纳农业与技术州立大学）
Sparse linear regression with ill-conditioned Gaussian random designs is widely believed to exhibit a statistical/computational gap, but there is surprisingly little formal evidence for this belief, even in the form of examples that are hard for restricted classes of algorithms. Recent work has shown that, for certain covariance matrices, the broad class of Preconditioned Lasso programs provably cannot succeed on polylogarithmically sparse signals with a sublinear number of samples. However, this lower bound only shows that for every preconditioner, there exists at least one signal that it fai…
Mean Field Variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is its (sometimes severe) underestimates of the uncertainty of model variables and lack of information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the …
Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation …
