Sparse signal recovery or compressed sensing can be formulated as certain sparse optimization problems. The classic optimization theory indicates that the Newton-like method often has a numerical advantage over the gradient method for nonlinear optimization problems. In this paper, we propose the so-called Newton-step-based iterative hard thresholding (NSIHT) and the Newton-step-based hard thresholding pursuit (NSHTP) algorithms for sparse signal recovery and signal approximation. Different from the traditional iterative hard thresholding (IHT) and hard thresholding pursuit (HTP), the proposed algorithms adopts the Newton-like search direction instead of the steepest descent direction.
A theoretical analysis for the proposed algorithms is carried out, and some sufficient conditions for the guaranteed success of sparse signal recovery via these algorithms are established. Our results are shown under the restricted isometry property which is one of the standard assumptions widely used in the field of compressed sensing and signal approximation. The empirical results obtained from synthetic data recovery indicate that the proposed algorithms are efficient signal recovery methods. The numerical stability of our algorithms in terms of the residual reduction is also investigated through simulations.
Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. …
In this paper, we will generate a convex iterative FP thresholding algorithm to solve the problem $(FP^λ_{a})$. Two schemes of convex iterative FP thresholding algorithms are generated. One is convex iterative FP thresholding algorithm-Scheme 1 and the other is convex iterative FP thresholding algorithm-Scheme 2. A global convergence theorem is proved for the convex iterative FP thresholding algorithm-Scheme 1. Under an adaptive rule, the convex iterative FP thresholding algorithm-Scheme 2 will be adaptive both for the choice of the regularized parameter $λ$ and parameter $a$. These are the ad…
The problem of recovering a low-rank matrix from the linear constraints, known as affine matrix rank minimization problem, has been attracting extensive attention in recent years. In general, affine matrix rank minimization problem is a NP-hard. In our latest work, a non-convex fraction function is studied to approximate the rank function in affine matrix rank minimization problem and translate the NP-hard affine matrix rank minimization problem into a transformed affine matrix rank minimization problem. A scheme of iterative singular value thresholding algorithm is generated to solve the regu…
Solutions to the linear complementarity problem (LCP) are naturally sparse in many applications such as bimatrix games and portfolio section problems. Despite that it gives rise to the hardness, sparsity makes optimization faster and enables relatively large scale computation. Motivated by this, we take the sparse LCP into consideration, investigating the existence and boundedness of its solution set as well as introducing a new merit function, which allows us to convert the problem into a sparsity constrained optimization. The function turns out to be continuously differentiable and twice con…
In this paper, we present modifications of the iterative hard thresholding (IHT) method for recovery of jointly row-sparse and low-rank matrices. In particular a Riemannian version of IHT is considered which significantly reduces computational cost of the gradient projection in the case of rank-one measurement operators, which have concrete applications in blind deconvolution. Experimental results are reported that show near-optimal recovery for Gaussian and rank-one measurements, and that adaptive stepsizes give crucial improvement. A Riemannian proximal gradient method is derived for the spe…
In this paper, we address the problem of distributed sparse recovery of signals acquired via compressed measurements in a sensor network. We propose a new class of distributed algorithms to solve Lasso regression problems, when the communication to a fusion center is not possible, e.g., due to communication cost or privacy reasons. More precisely, we introduce a distributed iterative soft thresholding algorithm (DISTA) that consists of three steps: an averaging step, a gradient step, and a soft thresholding operation. We prove the convergence of DISTA in networks represented by regular graphs,…
Hard-thresholding-based algorithms have seen various advantages for sparse optimization in controlling the sparsity and allowing for fast computation. Recent research shows that when techniques of the Newton-type methods are integrated, their numerical performance can be improved surprisingly. This paper develops a gradient projection Newton pursuit algorithm that mainly adopts the hard-thresholding operator and employs the Newton pursuit only when certain conditions are satisfied. The proposed algorithm is capable of converging globally and quadratically under the standard assumptions. When i…
Recently, the $ł_{p}$-norm regularization minimization problem $(P_{p}^λ)$ has attracted great attention in compressed sensing. However, the $ł_{p}$-norm $\|x\|_{p}^{p}$ in problem $(P_{p}^λ)$ is nonconvex and non-Lipschitz for all $p\in(0,1)$, and there are not many optimization theories and methods are proposed to solve this problem. In fact, it is NP-hard for all $p\in(0,1)$ and $λ>0$. In this paper, we study two modified $ł_{p}$ regularization minimization problems to approximate the NP-hard problem $(P_{p}^λ)$. Inspired by the good performance of Half algorithm and $2/3$ algorithm in some…
A fast iterative thresholding and support-and-scale shrinking algorithm (fits3) for non-lipschitz group sparse optimization (i): the case of least-squares fidelity
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
In this paper we consider the problem of exact recovery of a fixed sparse vector with the measurement matrices sequentially arriving along with corresponding measurements. We propose an extension of the iterative hard thresholding (IHT) algorithm, termed as sequential IHT (SIHT) which breaks the total time horizon into several phases such that IHT is executed in each of these phases using a fixed measurement matrix obtained at the beginning of that phase. We consider a stochastic setting where the measurement matrices obtained at each phase are independent samples of a sub Gaussian random matr…
The problem of recovering a low $n$-rank tensor is an extension of sparse recovery problem from the low dimensional space (matrix space) to the high dimensional space (tensor space) and has many applications in computer vision and graphics such as image inpainting and video inpainting. In this paper, we consider a new tensor recovery model, named as minimum $n$-rank approximation (MnRA), and propose an appropriate iterative hard thresholding algorithm with giving the upper bound of the $n$-rank in advance. The convergence analysis of the proposed algorithm is also presented. Particularly, we s…
We know that compressive sensing can establish stable sparse recovery results from highly undersampled data under a restricted isometry property condition. In reality, however, numerous problems are coherent, and vast majority conventional methods might work not so well. Recently, it was shown that using the difference between $\ell_1$- and $\ell_2$-norm as a regularization always has superior performance. In this paper, we propose an adaptive $\ell_p$-$\ell_{1-2}$ model where the $\ell_p$-norm with $p\geq 1$ measures the data fidelity and the $\ell_{1-2}$-term measures the sparsity. This prop…
Iterative hard thresholding (IHT) and compressive sampling matching pursuit (CoSaMP) are two types of mainstream compressed sensing algorithms using hard thresholding operators for signal recovery and approximation. The guaranteed performance for signal recovery via these algorithms has mainly been analyzed under the condition that the restricted isometry constant of a sensing matrix, denoted by $ δ_K$ (where $K$ is an integer number), is smaller than a certain threshold value in the interval $(0,1).$ The condition $ δ_{K}< δ^*$ for some constant $ δ^* \leq 1 $ ensuring the success of signal r…
In this paper, we consider the $\ell_0$ minimization problem whose objective function is the sum of $\ell_0$-norm and convex differentiable function. A variable metric type method which combines the PIHT method and the skill in quasi-newton method, named variable metric extrapolation proximal iterative hard thresholding (VMEPIHT) method, is proposed. Then we analyze its convergence, linear convergence rate and superlinear convergence rate under appropriate assumptions. Finally, we conduct numerical experiments on compressive sensing problem and CT image reconstruction problem to confirm VMPIHT…
The optimization problems with a sparsity constraint is a class of important global optimization problems. A typical type of thresholding algorithms for solving such a problem adopts the traditional full steepest descent direction or Newton-like direction as a search direction to generate an iterate on which a certain thresholding is performed. Traditional hard thresholding discards a large part of a vector when the vector is dense. Thus a large part of important information contained in a dense vector has been lost in such a thresholding process. Recent study [Zhao, SIAM J Optim, 30(1), pp. 3…
In recent studies on sparse modeling, non-convex penalties have received considerable attentions due to their superiorities on sparsity-inducing over the convex counterparts. Compared with the convex optimization approaches, however, the non-convex approaches have more challenging convergence analysis. In this paper, we study the convergence of a non-convex iterative thresholding algorithm for solving sparse recovery problems with a certain class of non-convex penalties, whose corresponding thresholding functions are discontinuous with jump discontinuities. Therefore, we call the algorithm the…
The optimal $k$-thresholding (OT) and optimal $k$-thresholding pursuit (OTP) are newly introduced frameworks of thresholding techniques for compressed sensing and signal approximation. Such frameworks motivate the practical and efficient algorithms called relaxed optimal $k$-thresholding ($\textrm{ROT}ω$) and relaxed optimal $k$-thresholding pursuit ($\textrm{ROTP}ω$) which are developed through the tightest convex relaxations of OT and OTP, where $ω$ is a prescribed integer number. The preliminary numerical results demonstrated in \cite{Z19} indicate that these approaches can stably reconstru…
We study a non-convex low-rank promoting penalty function, the transformed Schatten-1 (TS1), and its applications in matrix completion. The TS1 penalty, as a matrix quasi-norm defined on its singular values, interpolates the rank and the nuclear norm through a nonnegative parameter a. We consider the unconstrained TS1 regularized low-rank matrix recovery problem and develop a fixed point representation for its global minimizer. The TS1 thresholding functions are in closed analytical form for all parameter values. The TS1 threshold values differ in subcritical (supercritical) parameter regime w…
We propose and study a class of novel algorithms that aim at solving bilinear and quadratic inverse problems. Using a convex relaxation based on tensorial lifting, and applying first-order proximal algorithms, these problems could be solved numerically by singular value thresholding methods. However, a direct realization of these algorithms for, e.g., image recovery problems is often impracticable, since computations have to be performed on the tensor-product space, whose dimension is usually tremendous. To overcome this limitation, we derive tensor-free versions of common singular value thres…
In compressed sensing, the l0-norm minimization of sparse signal reconstruction is NP-hard. Recent work shows that compared with the best convex relaxation (l1-norm), nonconvex penalties can better approximate the l0-norm and can reconstruct the signal based on fewer observations. In this paper, the original problem is relaxed by using minimax concave penalty (MCP). Then alternating direction method of multipliers (ADMM) and modified iterative hard thresholding method are used to solve the problem. Under certain reasonable assumptions, the global convergence of the proposed method is proved. T…
University of Siegen（锡根大学）；University of Bonn（波恩大学）；Technical University of Munich（慕尼黑工业大学）
The idea of compressed sensing is to exploit representations in suitable (overcomplete) dictionaries that allow to recover signals far beyond the Nyquist rate provided that they admit a sparse representation in the respective dictionary. The latter gives rise to the sparse recovery problem of finding the best sparse linear approximation of given data in a given generating system. In this paper we analyze the iterative hard thresholding (IHT) algorithm as one of the most popular greedy methods for solving the sparse recovery problem, and demonstrate that systematically perturbing the IHT algori…
We recover jump-sparse and sparse signals from blurred incomplete data corrupted by (possibly non-Gaussian) noise using inverse Potts energy functionals. We obtain analytical results (existence of minimizers, complexity) on inverse Potts functionals and provide relations to sparsity problems. We then propose a new optimization method for these functionals which is based on dynamic programming and the alternating direction method of multipliers (ADMM). A series of experiments shows that the proposed method yields very satisfactory jump-sparse and sparse reconstructions, respectively. We highlig…
Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent
Department of Statistics and Data Science and Department of Computer Science, University of Central Florida（统计与数据科学系和计算机科学系，中央佛罗里达大学）；Department of Computational Mathematics, Science and Engineering and Department of Mathematics, Michigan State University（计算数学、科学与工程系和数学系，密歇根州立大学）；School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University（数学与统计学系，和湖北计算科学重点实验室，武汉大学）；School of Artificial Intelligence, Wuhan University, National Center for Applied Mathematics in Hubei, and Hubei Key Laboratory of Computational Science, Wuhan, China（人工智能学院，武汉大学，湖北应用数学中心，和湖北计算科学重点实验室，中国武汉）；Institute for Advanced Study, Shenzhen University（先进研究学院，深圳大学）
The sparse inverse covariance estimation problem is commonly solved using an $\ell_{1}$-regularized Gaussian maximum likelihood estimator known as "graphical lasso", but its computational cost becomes prohibitive for large data sets. A recent line of results showed--under mild assumptions--that the graphical lasso estimator can be retrieved by soft-thresholding the sample covariance matrix and solving a maximum determinant matrix completion (MDMC) problem. This paper proves an extension of this result, and describes a Newton-CG algorithm to efficiently solve the MDMC problem. Assuming that the…
The problem of the distributed recovery of jointly sparse signals has attracted much attention recently. Let us assume that the nodes of a network observe different sparse signals with common support; starting from linear, compressed measurements, and exploiting network communication, each node aims at reconstructing the support and the non-zero values of its observed signal. In the literature, distributed greedy algorithms have been proposed to tackle this problem, among which the most reliable ones require a large amount of transmitted data, which barely adapts to realistic network communica…
School of Electrical and Computer Engineering（电气与计算机工程学院）；Georgia Institute of Technology（佐治亚理工学院）；Department of Electrical Engineering（电气工程系）；Stanford University（斯坦福大学）
We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients ($n \gg p \gg 1$). In this regime, optimization algorithms can immensely benefit from approximate second order information. We propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a Stein-type lemma, which allows further improvements through sub-sampling and eigenvalue thresholding. Our algorithm enjoys fast convergence rates, re…
We consider algorithms and recovery guarantees for the analysis sparse model in which the signal is sparse with respect to a highly coherent frame. We consider the use of a monotone version of the fast iterative shrinkage- thresholding algorithm (MFISTA) to solve the analysis sparse recovery problem. Since the proximal operator in MFISTA does not have a closed-form solution for the analysis model, it cannot be applied directly. Instead, we examine two alternatives based on smoothing and decomposition transformations that relax the original sparse recovery problem, and then implement MFISTA on …
Alternating Iteratively Reweighted $\ell_1$ and Subspace Newton Algorithms for Nonconvex Sparse Optimization
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；School of Mathematics and Statistics, Henan University（河南大学数学与统计学院）；Center for Applied Mathematics of Henan Province, Henan University（河南大学河南省应用数学中心）
In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including $\ell^0$, bridge, smoothly clipped absolute deviation, capped $\ell^1$ and minimax concavity penalty. First we establish the existence of a global minimizer for the related optimization problems. Then we derive a novel necessary optimality condition for the global minimizer using the associated thresholding operator. The solutions to the optimality system a…
Non-smooth regularization is widely used in image reconstruction to eliminate the noise while preserving subtle image structures. In this work, we investigate the use of proximal Newton (PN) method to solve an optimization problem with a smooth data-fidelity term and total variation (TV) regularization arising from image reconstruction applications. Specifically, we consider a nonlinear Poisson-modeled single-energy X-ray computed tomography reconstruction problem with the data-fidelity term given by the I-divergence. The PN algorithm is compared to state-of-the-art first-order proximal algori…
