In this paper, we aim to give a theoretical approximation for the penalty level of $\ell_{1}$-regularization problems. This can save much time in practice compared with the traditional methods, such as cross-validation. To achieve this goal, we develop two Gaussian approximation methods, which are based on a moderate deviation theorem and Stein's method respectively. Both of them give efficient approximations and have good performances in simulations. We apply the two Gaussian approximation methods into three types of ultra-high dimensional $\ell_{1}$ penalized regressions: lasso, square-root lasso, and weighted $\ell_{1}$ penalized Poisson regression. The numerical results indicate that our two ways to estimate the penalty levels achieve high computational efficiency. Besides, our prediction errors outperform that based on the 10-fold cross-validation.
National University of Singapore（新加坡国立大学）；Nanjing University of Science and Technology（南京理工大学）；Peking University（北京大学）
As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex low-rank minimization problem is much more challenging than the nonconvex sparse minimization problem. We observe that all the existing nonconvex penalty functions are concave and monotonically increasing on $[0,\infty)$. Thus their gradients are decreasing functions. Based on thi…
Sparse clustering, which aims to find a proper partition of an extremely high-dimensional data set with redundant noise features, has been attracted more and more interests in recent years. The existing studies commonly solve the problem in a framework of maximizing the weighted feature contributions subject to a $\ell_2/\ell_1$ penalty. Nevertheless, this framework has two serious drawbacks: One is that the solution of the framework unavoidably involves a considerable portion of redundant noise features in many situations, and the other is that the framework neither offers intuitive explanati…
We propose novel necessary and sufficient conditions for a sensing matrix to be "$s$-good" - to allow for exact $\ell_1$-recovery of sparse signals with $s$ nonzero entries when no measurement noise is present. Then we express the error bounds for imperfect $\ell_1$-recovery (nonzero measurement noise, nearly $s$-sparse signal, near-optimal solution of the optimization problem yielding the $\ell_1$-recovery) in terms of the characteristics underlying these conditions. Further, we demonstrate (and this is the principal result of the paper) that these characteristics, although difficult to evalu…
Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty
Institute of Applied Mathematics, Shenzhen Polytechnic University（深圳职业技术学院应用数学研究所）；School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；Center for Applied Mathematics of Henan Province, Henan University（河南大学河南省应用数学中心）
We investigate properties of estimators obtained by minimization of U-processes with the Lasso penalty in high-dimensional settings. Our attention is focused on the ranking problem that is popular in machine learning. It is related to guessing the ordering between objects on the basis of their observed predictors. We prove the oracle inequality for the excess risk of the considered estimator as well as the bound for the l1 distance between the estimator and the oracle.
Structured sparsity is an important part of the modern statistical toolkit. We say a set of model parameters has block diagonal sparsity up to permutations if its elements can be viewed as the edges of a graph that has multiple connected components. For example, a block diagonal correlation matrix with K blocks of variables corresponds to a graph with K connected components whose nodes are the variables and whose edges are the correlations. This type of sparsity captures clusters of model parameters. To learn block diagonal sparsity patterns we develop the folded concave Laplacian spectral pen…
In many application areas we are faced with the following question: Can we recover a sparse vector $x_o \in \mathbb{R}^N$ from its undersampled set of noisy observations $y \in \mathbb{R}^n$, $y=A x_o+w$. The last decade has witnessed a surge of algorithms and theoretical results addressing this question. One of the most popular algorithms is the $\ell_p$-regularized least squares (LPLS) given by the following formulation: \[ \hat{x}(γ,p )\in \arg\min_x \frac{1}{2}\|y - Ax\|_2^2+γ\|x\|_p^p, \] where $p \in [0,1]$. Despite the non-convexity of these problems for $p<1$, they are still appealing …
P-splines are penalized B-splines, in which finite order differences in coefficients are typically penalized with an $\ell_2$ norm. P-splines can be used for semiparametric regression and can include random effects to account for within-subject variability. In addition to $\ell_2$ penalties, $\ell_1$-type penalties have been used in nonparametric and semiparametric regression to achieve greater flexibility, such as in locally adaptive regression splines, $\ell_1$ trend filtering, and the fused lasso additive model. However, there has been less focus on using $\ell_1$ penalties in P-splines, pa…
In this paper, we propose a majorization-minimization (MM) algorithm for high-dimensional fused lasso regression (FLR) suitable for parallelization using graphics processing units (GPUs). The MM algorithm is stable and flexible as it can solve the FLR problems with various types of design matrices and penalty structures within a few tens of iterations. We also show that the convergence of the proposed algorithm is guaranteed. We conduct numerical studies to compare our algorithm with other existing algorithms, demonstrating that the proposed MM algorithm is competitive in many settings includi…
Total Variation, Adaptive Total Variation and Nonconvex Smoothly Clipped Absolute Deviation Penalty for Denoising Blocky Images
School of Computing Sciences, VIT University, Vellore, TN, India（计算机科学学院，VIT大学，Vellore，印度）；Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore, 637371（数学科学系，南洋理工大学，新加坡）
We investigate the difference between using an $\ell_1$ penalty versus an $\ell_1$ constraint in generalized eigenvalue problems, such as principal component analysis and discriminant analysis. Our main finding is that an $\ell_1$ penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an $\ell_1$ constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of an $\ell_1$ constraint in the context of discriminant analysis and principal component analysis.
We tackle the problem of recovering an unknown signal observed in an ill-posed inverse problem framework. More precisely, we study a procedure commonly used in numerical analysis or image deblurring: minimizing an empirical loss function balanced by an $l^1$ penalty, acting as a sparsity constraint. We prove that, by choosing a proper loss function, this estimation technique enables to build an adaptive estimator, in the sense that it converges at the optimal rate of convergence without prior knowledge of the regularity of the true solution
We study high-dimensional estimators with the trimmed $\ell_1$ penalty, which leaves the $h$ largest parameter entries penalty-free. While optimization techniques for this nonconvex penalty have been studied, the statistical properties have not yet been analyzed. We present the first statistical analyses for $M$-estimation and characterize support recovery, $\ell_\infty$ and $\ell_2$ error of the trimmed $\ell_1$ estimates as a function of the trimming parameter $h$. Our results show different regimes based on how $h$ compares to the true support size. Our second contribution is a new algorith…
This paper concerns solving the sparse deconvolution and demixing problem using $\ell_{1,2}$-minimization. We show that under a certain structured random model, robust and stable recovery is possible. The results extend results of Ling and Strohmer [Self Calibration and Biconvex Compressive Sensing, Inverse Problems, 2015], and in particular theoretically explain certain experimental findings from that paper. Our results do not only apply to the deconvolution and demixing problem, but to recovery of column-sparse matrices in general.
In the paper, we proposed the Dantzig selector based on the $\ell_{1}-α\ell_{2}$~$(0< α\leq1)$ minimization for the signal recovery. In the Dantzig selector, the constraint $\|{\bf A}^{\top}({\bf b}-{\bf A}{\bf x})\|_\infty \leq η$ for some small constant $η>0$ means the columns of ${\bf A}$ has very weakly correlated with the error vector ${\bf e}={\bf A}{\bf x}-{\bf b}$. First, recovery guarantees based on the restricted isometry property (RIP) are established for signals. Next, we propose the effective algorithm to solve the proposed Dantzig selector. Last, we illustrate the proposed model …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
We study the robust mean estimation problem in high dimensions, where $α<0.5$ fraction of the data points can be arbitrarily corrupted. Motivated by compressive sensing, we formulate the robust mean estimation problem as the minimization of the $\ell_0$-`norm' of the outlier indicator vector, under second moment constraints on the inlier data points. We prove that the global minimum of this objective is order optimal for the robust mean estimation problem, and we propose a general framework for minimizing the objective. We further leverage the $\ell_1$ and $\ell_p$ $(0<p<1)$, minimization tech…
