Given an unknown signal $\mathbf{x}_0\in\mathbb{R}^n$ and linear noisy measurements $\mathbf{y}=\mathbf{A}\mathbf{x}_0+σ\mathbf{v}\in\mathbb{R}^m$, the generalized $\ell_2^2$-LASSO solves $\hat{\mathbf{x}}:=\arg\min_{\mathbf{x}}\frac{1}{2}\|\mathbf{y}-\mathbf{A}\mathbf{x}\|_2^2 + σλf(\mathbf{x})$. Here, $f$ is a convex regularization function (e.g. $\ell_1$-norm, nuclear-norm) aiming to promote the structure of $\mathbf{x}_0$ (e.g. sparse, low-rank), and, $λ\geq 0$ is the regularizer parameter. A related optimization problem, though not as popular or well-known, is often referred to as the generalized $\ell_2$-LASSO and takes the form $\hat{\mathbf{x}}:=\arg\min_{\mathbf{x}}\|\mathbf{y}-\mathbf{A}\mathbf{x}\|_2 + λf(\mathbf{x})$, and has been analyzed in [1]. [1] further made conjectures about the performance of the generalized $\ell_2^2$-LASSO. This paper establishes these conjectures rigorously. We measure performance with the normalized squared error $\mathrm{NSE}(σ):=\|\hat{\mathbf{x}}-\mathbf{x}_0\|_2^2/σ^2$. Assuming the entries of $\mathbf{A}$ and $\mathbf{v}$ be i.i.d. standard normal, we precisely characterize the "asymptotic NSE" $\mathrm{aNSE}:=\lim_{σ\rightarrow 0}\mathrm{NSE}(σ)$ when the problem dimensions $m,n$ tend to infinity in a proportional manner. The role of $λ,f$ and $\mathbf{x}_0$ is explicitly captured in the derived expression via means of a single geometric quantity, the Gaussian distance to the subdifferential. We conjecture that $\mathrm{aNSE} = \sup_{σ>0}\mathrm{NSE}(σ)$. We include detailed discussions on the interpretation of our result, make connections to relevant literature and perform computational experiments that validate our theoretical findings.
In this paper we consider the conservative Lasso which we argue penalizes more correctly than the Lasso and show how it may be desparsified in the sense of van de Geer et al. (2014) in order to construct asymptotically honest (uniform) confidence bands. In particular, we develop an oracle inequality for the conservative Lasso only assuming the existence of a certain number of moments. This is done by means of the Marcinkiewicz-Zygmund inequality. We allow for heteroskedastic non-subgaussian error terms and covariates. Next, we desparsify the conservative Lasso estimator and derive the asymptot…
We consider a multivariate finite mixture of Gaussian regression models for high-dimensional data, where the number of covariates and the size of the response may be much larger than the sample size. We provide an $\ell_1$-oracle inequality satisfied by the Lasso estimator according to the Kullback-Leibler loss. This result is an extension of the $\ell_1$-oracle inequality established by Meynet in \cite{Meynet} in the multivariate case. We focus on the Lasso for its $\ell_1$-regularization properties rather than for the variable selection procedure, as it was done in Städler in \cite{Stadler}.
In this paper, we are concerned with regularized regression problems where the prior regularizer is a proper lower semicontinuous and convex function which is also partly smooth relative to a Riemannian submanifold. This encompasses as special cases several known penalties such as the Lasso ($\ell^1$-norm), the group Lasso ($\ell^1-\ell^2$-norm), the $\ell^\infty$-norm, and the nuclear norm. This also includes so-called analysis-type priors, i.e. composition of the previously mentioned penalties with linear operators, typical examples being the total variation or fused Lasso penalties.We study…
This paper considers the problem of recovering a group sparse signal matrix $\mathbf{Y} = [\mathbf{y}_1, \cdots, \mathbf{y}_L]$ from sparsely corrupted measurements $\mathbf{M} = [\mathbf{A}_{(1)}\mathbf{y}_{1}, \cdots, \mathbf{A}_{(L)}\mathbf{y}_{L}] + \mathbf{S}$, where $\mathbf{A}_{(i)}$'s are known sensing matrices and $\mathbf{S}$ is an unknown sparse error matrix. A robust group lasso (RGL) model is proposed to recover $\mathbf{Y}$ and $\mathbf{S}$ through simultaneously minimizing the $\ell_{2,1}$-norm of $\mathbf{Y}$ and the $\ell_1$-norm of $\mathbf{S}$ under the measurement constrain…
We consider statistical inference for a single coordinate of regression coefficients in high-dimensional linear models. Recently, the debiased estimators are popularly used for constructing confidence intervals and hypothesis testing in high-dimensional models. However, some representative numerical experiments show that they tend to be biased for large coefficients, especially when the number of large coefficients dominates the number of small coefficients. In this paper, we propose a modified debiased Lasso estimator based on bootstrap. Let us denote the proposed estimator BS-DB for short. W…
A fundamental problem in the high-dimensional regression is to understand the tradeoff between type I and type II errors or, equivalently, false discovery rate (FDR) and power in variable selection. To address this important problem, we offer the first complete tradeoff diagram that distinguishes all pairs of FDR and power that can be asymptotically realized by the Lasso with some choice of its penalty parameter from the remaining pairs, in a regime of linear sparsity under random designs. The tradeoff between the FDR and power characterized by our diagram holds no matter how strong the signal…
Consider estimating an unknown, but structured, signal $x_0\in R^n$ from $m$ measurement $y_i=g_i(a_i^Tx_0)$, where the $a_i$'s are the rows of a known measurement matrix $A$, and, $g$ is a (potentially unknown) nonlinear and random link-function. Such measurement functions could arise in applications where the measurement device has nonlinearities and uncertainties. It could also arise by design, e.g., $g_i(x)=\text{sign}(x+z_i)$, corresponds to noisy 1-bit quantized measurements. Motivated by the classical work of Brillinger, and more recent work of Plan and Vershynin, we estimate $x_0$ via …
A classical problem that arises in numerous signal processing applications asks for the reconstruction of an unknown, $k$-sparse signal $x_0\in R^n$ from underdetermined, noisy, linear measurements $y=Ax_0+z\in R^m$. One standard approach is to solve the following convex program $\hat x=\arg\min_x \|y-Ax\|_2 + λ\|x\|_1$, which is known as the $\ell_2$-LASSO. We assume that the entries of the sensing matrix $A$ and of the noise vector $z$ are i.i.d Gaussian with variances $1/m$ and $σ^2$. In the large system limit when the problem dimensions grow to infinity, but in constant rates, we \emph{pre…
We propose a convex formulation of the fused lasso signal approximation problem consisting of non-convex penalty functions. The fused lasso signal model aims to estimate a sparse piecewise constant signal from a noisy observation. Originally, the $\ell_1$ norm was used as a sparsity-inducing convex penalty function for the fused lasso signal approximation problem. However, the $\ell_1$ norm underestimates signal values. Non-convex sparsity-inducing penalty functions better estimate signal values. In this paper, we show how to ensure the convexity of the fused lasso signal approximation problem…
In this paper, we study the sharp oracle bounds for Slope and Lasso and generalize the results in Bellec et al. (2018) to allow the case that the parameter vector is not exactly sparse and obtain the optimal bounds for $\ell_q$ estimation errors with $1\leq q\leq \infty$ by using some extended Restricted Eigenvalue type conditions.
We show that two polynomial time methods, a Lasso estimator with adaptively chosen tuning parameter and a Slope estimator, adaptively achieve the exact minimax prediction and $\ell_2$ estimation rate $(s/n)\log (p/s)$ in high-dimensional linear regression on the class of $s$-sparse target vectors in $\mathbb R^p$. This is done under the Restricted Eigenvalue (RE) condition for the Lasso and under a slightly more constraining assumption on the design for the Slope. The main results have the form of sharp oracle inequalities accounting for the model misspecification error. The minimax optimal bo…
This paper proposes a general framework for penalized convex empirical criteria and a new version of the Sparse-Group LASSO (SGL, Simon and al., 2013), called the adaptive SGL, where both penalties of the SGL are weighted by preliminary random coefficients. We explore extensively its asymptotic properties and prove that this estimator satisfies the so-called oracle property (Fan and Li, 2001), that is the sparsity based estimator recovers the true underlying sparse model and is asymptotically normally distributed. Then we study its asymptotic properties in a double-asymptotic framework, where …
We study a high-dimensional regression model. Aim is to construct a confidence set for a given group of regression coefficients, treating all other regression coefficients as nuisance parameters. We apply a one-step procedure with the square-root Lasso as initial estimator and a multivariate square-root Lasso for constructing a surrogate Fisher information matrix. The multivariate square-root Lasso is based on nuclear norm loss with $\ell_1$-penalty. We show that this procedure leads to an asymptotically $χ^2$-distributed pivot, with a remainder term depending only on the $\ell_1$-error of the…
The LASSO is a variable subset selection procedure in statistical linear regression based on $\ell_1$ penalization of the least-squares operator. Uniqueness of the LASSO is an important issue, especially for the study of the LASSO path. The goal of the present paper is to provide a generic sufficient condition on the design matrix for the LASSO minimizer to be unique. Unlike previous works on the question of uniqueness, our condition only depends on the design matrix. Our study is based on a general position condition on the design matrix which holds with probability one for most experimental …
Basis Pursuit (BP), Basis Pursuit DeNoising (BPDN), and LASSO are popular methods for identifying important predictors in the high-dimensional linear regression model, i.e. when the number of rows of the design matrix X is smaller than the number of columns. By definition, BP uniquely recovers the vector of regression coefficients b if there is no noise and the vector b has the smallest L1 norm among all vectors s such that Xb=Xs (identifiability condition). Furthermore, LASSO can recover the sign of b only under a much stronger irrepresentability condition. Meanwhile, it is known that the mod…
This paper proposes a theory for $\ell_1$-norm penalized high-dimensional $M$-estimators, with nonconvex risk and unrestricted domain. Under high-level conditions, the estimators are shown to attain the rate of convergence $s_0\sqrt{\log(nd)/n}$, where $s_0$ is the number of nonzero coefficients of the parameter of interest. Sufficient conditions for our main assumptions are then developed and finally used in several examples including robust linear regression, binary classification and nonlinear least squares.
In this paper, we aim to give a theoretical approximation for the penalty level of $\ell_{1}$-regularization problems. This can save much time in practice compared with the traditional methods, such as cross-validation. To achieve this goal, we develop two Gaussian approximation methods, which are based on a moderate deviation theorem and Stein's method respectively. Both of them give efficient approximations and have good performances in simulations. We apply the two Gaussian approximation methods into three types of ultra-high dimensional $\ell_{1}$ penalized regressions: lasso, square-root …
Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection
Department of Statistics, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校统计系）；Department of Electrical and Computer Engineering, the Grainger College of Engineering（格拉inger工程学院电子与计算机工程系）
The University of Tokyo（东京大学）；Kyoto University（京都大学）
We theoretically analyze the model selection consistency of least absolute shrinkage and selection operator (Lasso), both with and without post-thresholding, for high-dimensional Ising models. For random regular (RR) graphs of size $p$ with regular node degree $d$ and uniform couplings $θ_0$, it is rigorously proved that Lasso \textit{without post-thresholding} is model selection consistent in the whole paramagnetic phase with the same order of sample complexity $n=Ω{(d^3\log{p})}$ as that of $\ell_1$-regularized logistic regression ($\ell_1$-LogR). This result is consistent with the conjectur…
In a linear regression model of fixed dimension $p \leq n$, we construct confidence regions for the unknown parameter vector based on the Lasso estimator that uniformly and exactly hold the prescribed in finite samples as well as in an asymptotic setup. We thereby quantify estimation uncertainty as well as the "post-model selection error" of this estimator. More concretely, in finite samples with Gaussian errors and asymptotically in the case where the Lasso estimator is tuned to perform conservative model selection, we derive exact formulas for computing the minimal coverage probability over …
We study uniqueness in the generalized lasso problem, where the penalty is the $\ell_1$ norm of a matrix $D$ times the coefficient vector. We derive a broad result on uniqueness that places weak assumptions on the predictor matrix $X$ and penalty matrix $D$; the implication is that, if $D$ is fixed and its null space is not too large (the dimension of its null space is at most the number of samples), and $X$ and response vector $y$ jointly follow an absolutely continuous distribution, then the generalized lasso problem has a unique solution almost surely, regardless of the number of predictors…
Many data sets consist of variables with an inherent group structure. The problem of group selection has been well studied, but in this paper, we seek to do the opposite: our goal is to select at least one variable from each group in the context of predictive regression modeling. This problem is NP-hard, but we study the tightest convex relaxation: a composite penalty that is a combination of the $\ell_1$ and $\ell_2$ norms. Our so-called Exclusive Lasso method performs structured variable selection by ensuring that at least one variable is selected from each group. We study our method's stati…
Sparse Group LASSO (SGL) is a regularized model for high-dimensional linear regression problems with grouped covariates. SGL applies $l_1$ and $l_2$ penalties on the individual predictors and group predictors, respectively, to guarantee sparse effects both on the inter-group and within-group levels. In this paper, we apply the approximate message passing (AMP) algorithm to efficiently solve the SGL problem under Gaussian random designs. We further use the recently developed state evolution analysis of AMP to derive an asymptotically exact characterization of SGL solution. This allows us to con…
The Lasso is one of the most important approaches for parameter estimation and variable selection in high dimensional linear regression. At the heart of its success is the attractive rate of convergence result even when $p$, the dimension of the problem, is much larger than the sample size $n$. In particular, Bickel et al. (2009) showed that this rate, in terms of the $\ell_1$ norm, is of the order $s\sqrt{(\log p)/n}$ for a sparsity index $s$. In this paper, we obtain a new bound on the convergence rate by taking advantage of the distributional information of the model. Under the normality or…
Selection of important covariates and to drop the unimportant ones from a high-dimensional regression model is a long standing problem and hence have received lots of attention in the last two decades. After selecting the correct model, it is also important to properly estimate the existing parameters corresponding to important covariates. In this spirit, Fan and Li (2001) proposed Oracle property as a desired feature of a variable selection method. Oracle property has two parts; one is the variable selection consistency (VSC) and the other one is the asymptotic normality. Keeping VSC fixed an…
University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）；Central China Normal University（华中师范大学）
When we are interested in high-dimensional system and focus on classification performance, the $\ell_{1}$-penalized logistic regression is becoming important and popular. However, the Lasso estimates could be problematic when penalties of different coefficients are all the same and not related to the data. We proposed two types of weighted Lasso estimates depending on covariates by the McDiarmid inequality. Given sample size $n$ and dimension of covariates $p$, the finite sample behavior of our proposed methods with a diverging number of predictors is illustrated by non-asymptotic oracle inequ…
Some Two-Step Procedures for Variable Selection in High-Dimensional Linear Regression
我们研究了高维变量选择问题，通过一些两个步骤的程序。首先，我们表明，给定一些好的初始估计，这是$\ell_{\infty}$ -一致的，但不一定变量选择一致的，我们可以应用非负Garrote，自适应Lasso或硬阈值的过程，以获得一个最终的估计，估计和变量选择一致。与Lasso不同，我们的结果不需要不可表示的条件，即使对于中度$p_n$也可能容易失败（Zhao和Yu，2007），并且它还允许$p_n$几乎与$\exp(n)$一样快地生长（对于硬阈值，对$p_n$没有限制）。我们还研究了岭回归可以用作初始估计的条件。我们证明了在放松的可识别条件下，岭估计是$\ell_{\infty}$ -相容的.当使用$p_n\le n$时，通常满足这样的条件，并且不需要相关和不相关协变量之间的部分正交性，而这是单变量回归所需的（Huang et al.，2008年）。我们的数值研究表明，当使用Lasso或Ridge作为初始估计量时，两步程序具有比使用单变量回归的Lasso或自适应Lasso更高的稀疏恢复率（Huang et al.，2008年）。
