The objective of this paper is to develop methods for solving image recovery problems subject to constraints on the solution. More precisely, we will be interested in problems which can be formulated as the minimization over a closed convex constraint set of the sum of two convex functions f and g, where f may be non-smooth and g is differentiable with a Lipschitz-continuous gradient. To reach this goal, we derive two types of algorithms that combine forward-backward and Douglas-Rachford iterations. The weak convergence of the proposed algorithms is proved. In the case when the Lipschitz-continuity property of the gradient of g is not satisfied, we also show that, under some assumptions, it remains possible to apply these methods to the considered optimization problem by making use of a quadratic extension technique. The effectiveness of the algorithms is demonstrated for two wavelet-based image restoration problems involving a signal-dependent Gaussian noise and a Poisson noise, respectively.
In this paper, we will generate a convex iterative FP thresholding algorithm to solve the problem $(FP^λ_{a})$. Two schemes of convex iterative FP thresholding algorithms are generated. One is convex iterative FP thresholding algorithm-Scheme 1 and the other is convex iterative FP thresholding algorithm-Scheme 2. A global convergence theorem is proved for the convex iterative FP thresholding algorithm-Scheme 1. Under an adaptive rule, the convex iterative FP thresholding algorithm-Scheme 2 will be adaptive both for the choice of the regularized parameter $λ$ and parameter $a$. These are the ad…
The problem of recovering a low-rank matrix from the linear constraints, known as affine matrix rank minimization problem, has been attracting extensive attention in recent years. In general, affine matrix rank minimization problem is a NP-hard. In our latest work, a non-convex fraction function is studied to approximate the rank function in affine matrix rank minimization problem and translate the NP-hard affine matrix rank minimization problem into a transformed affine matrix rank minimization problem. A scheme of iterative singular value thresholding algorithm is generated to solve the regu…
We consider the inverse problem of recovering a binary function from blurred and noisy data. Such problems arise in many applications, for example image processing and optimal control of PDEs. Our formulation is based on the Mumford-Shah model, but with a phase field approximation to the perimeter regularisation. We use a double obstacle potential as well as a smooth double well potential. We introduce an iterative method for solving the problem, develop a suitable discretisation of this iterative method, and prove some convergence results. Numerical simulations are presented which illustrate …
Sparse signal recovery or compressed sensing can be formulated as certain sparse optimization problems. The classic optimization theory indicates that the Newton-like method often has a numerical advantage over the gradient method for nonlinear optimization problems. In this paper, we propose the so-called Newton-step-based iterative hard thresholding (NSIHT) and the Newton-step-based hard thresholding pursuit (NSHTP) algorithms for sparse signal recovery and signal approximation. Different from the traditional iterative hard thresholding (IHT) and hard thresholding pursuit (HTP), the proposed…
We present a new, simple and computationally efficient iterative method for low rank matrix completion. Our method is inspired by the class of factorization-type iterative algorithms, but substantially differs from them in the way the problem is cast. Precisely, given a target rank $r$, instead of optimizing on the manifold of rank $r$ matrices, we allow our interim estimated matrix to have a specific over-parametrized rank $2r$ structure. Our algorithm, denoted R2RILS for rank $2r$ iterative least squares, has low memory requirements, and at each iteration it solves a computationally cheap sp…
The Pennsylvania State University（宾夕法尼亚州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this letter, we address sparse signal recovery using spike and slab priors. In particular, we focus on a Bayesian framework where sparsity is enforced on reconstruction coefficients via probabilistic priors. The optimization resulting from spike and slab prior maximization is known to be a hard non-convex problem, and existing solutions involve simplifying assumptions and/or relaxations. We propose an approach called Iterative Convex Refinement (ICR) that aims to solve the aforementioned optimization problem directly allowing for greater generality in the sparse structure. Essentially, ICR …
Parabolic optimal control problems with control constraints are generally challenging, from either theoretical analysis or algorithmic design perspectives. Conceptually, the well-known alternating direction method of multipliers (ADMM) can be directly applied to such a problem. An attractive advantage of this direct ADMM application is that the control constraint can be untied from the parabolic PDE constraint; these two inherently different constraints thus can be treated individually in iterations. At each iteration of the ADMM, the main computation is for solving an unconstrained parabolic …
Recently, the $ł_{p}$-norm regularization minimization problem $(P_{p}^λ)$ has attracted great attention in compressed sensing. However, the $ł_{p}$-norm $\|x\|_{p}^{p}$ in problem $(P_{p}^λ)$ is nonconvex and non-Lipschitz for all $p\in(0,1)$, and there are not many optimization theories and methods are proposed to solve this problem. In fact, it is NP-hard for all $p\in(0,1)$ and $λ>0$. In this paper, we study two modified $ł_{p}$ regularization minimization problems to approximate the NP-hard problem $(P_{p}^λ)$. Inspired by the good performance of Half algorithm and $2/3$ algorithm in some…
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
In this paper we consider the problem of exact recovery of a fixed sparse vector with the measurement matrices sequentially arriving along with corresponding measurements. We propose an extension of the iterative hard thresholding (IHT) algorithm, termed as sequential IHT (SIHT) which breaks the total time horizon into several phases such that IHT is executed in each of these phases using a fixed measurement matrix obtained at the beginning of that phase. We consider a stochastic setting where the measurement matrices obtained at each phase are independent samples of a sub Gaussian random matr…
School of Electrical and Computer Engineering（电气与计算机工程学院）；Georgia Institute of Technology（佐治亚理工学院）；Department of Electrical Engineering（电气工程系）；Stanford University（斯坦福大学）
University of Chicago（芝加哥大学）；University of Southern California（南加州大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessian sketch is an optimization process with preconditioning, and develop accelerated iterative Hessian sketch via the searching the conjugate direction; we then establish primal-dual connections between the Hessian sketch and dual random projection, and apply the preconditioned conj…
The problem of recovering a low $n$-rank tensor is an extension of sparse recovery problem from the low dimensional space (matrix space) to the high dimensional space (tensor space) and has many applications in computer vision and graphics such as image inpainting and video inpainting. In this paper, we consider a new tensor recovery model, named as minimum $n$-rank approximation (MnRA), and propose an appropriate iterative hard thresholding algorithm with giving the upper bound of the $n$-rank in advance. The convergence analysis of the proposed algorithm is also presented. Particularly, we s…
In this paper, we present modifications of the iterative hard thresholding (IHT) method for recovery of jointly row-sparse and low-rank matrices. In particular a Riemannian version of IHT is considered which significantly reduces computational cost of the gradient projection in the case of rank-one measurement operators, which have concrete applications in blind deconvolution. Experimental results are reported that show near-optimal recovery for Gaussian and rank-one measurements, and that adaptive stepsizes give crucial improvement. A Riemannian proximal gradient method is derived for the spe…
Compressed sensing is a central topic in signal processing with myriad applications, where the goal is to recover a signal from as few observations as possible. Iterative re-weighting is one of the fundamental tools to achieve this goal. This paper re-examines the iteratively reweighted least squares (IRLS) algorithm for sparse recovery proposed by Daubechies, Devore, Fornasier, and Güntürk in \emph{Iteratively reweighted least squares minimization for sparse recovery}, {\sf Communications on Pure and Applied Mathematics}, {\bf 63}(2010) 1--38. Under the null space property of order $K$, the a…
An algorithmic framework, based on the difference of convex functions algorithm (DCA), is proposed for minimizing a class of concave sparse metrics for compressed sensing problems. The resulting algorithm iterates a sequence of $\ell_1$ minimization problems. An exact sparse recovery theory is established to show that the proposed framework always improves on the basis pursuit ($\ell_1$ minimization) and inherits robustness from it. Numerical examples on success rates of sparse solution recovery illustrate further that, unlike most existing non-convex compressed sensing solvers in the literatu…
Sparse recovery is one of the most fundamental and well-studied inverse problems. Standard statistical formulations of the problem are provably solved by general convex programming techniques and more practical, fast (nearly-linear time) iterative methods. However, these latter "fast algorithms" have previously been observed to be brittle in various real-world settings. We investigate the brittleness of fast sparse recovery algorithms to generative model changes through the lens of studying their robustness to a "helpful" semi-random adversary, a framework which tests whether an algorithm over…
We develop an adjoint approach for recovering the topographical function included in the source term of one-dimensional hyperbolic balance laws. We focus on a specific system, namely the shallow water equations, in an effort to recover the riverbed topography. The novelty of this work is the ability to robustly recover the bottom topography using only noisy boundary data from one measurement event and the inclusion of two regularization terms in the iterative update scheme. The adjoint scheme is determined from a linearization of the forward system and is used to compute the gradient of a cost…
Community detection in graphs that are generated according to stochastic block models (SBMs) has received much attention lately. In this paper, we focus on the binary symmetric SBM -- in which a graph of $n$ vertices is randomly generated by first partitioning the vertices into two equal-sized communities and then connecting each pair of vertices with probability that depends on their community memberships -- and study the associated exact community recovery problem. Although the maximum-likelihood formulation of the problem is non-convex and discrete, we propose to tackle it using a popular i…
We propose a new Iteratively Reweighted Least Squares (IRLS) algorithm for the problem of completing or denoising low-rank matrices that are structured, e.g., that possess a Hankel, Toeplitz or block-Hankel/Toeplitz structure. The algorithm optimizes an objective based on a non-convex surrogate of the rank by solving a sequence of quadratic problems. Our strategy combines computational efficiency, as it operates on a lower dimensional generator space of the structured matrices, with high statistical accuracy which can be observed in experiments on hard estimation and completion tasks. Our expe…
Low rank matrix recovery problems, including matrix completion and matrix sensing, appear in a broad range of applications. In this work we present GNMR -- an extremely simple iterative algorithm for low rank matrix recovery, based on a Gauss-Newton linearization. On the theoretical front, we derive recovery guarantees for GNMR in both the matrix sensing and matrix completion settings. Some of these results improve upon the best currently known for other methods. A key property of GNMR is that it implicitly keeps the factor matrices approximately balanced throughout its iterations. On the empi…
IIT Kanpur（印度理工学院坎普尔分校）；Microsoft Research India（微软印度研究院）
We provide the first global model recovery results for the IRLS (iteratively reweighted least squares) heuristic for robust regression problems. IRLS is known to offer excellent performance, despite bad initializations and data corruption, for several parameter estimation problems. Existing analyses of IRLS frequently require careful initialization, thus offering only local convergence guarantees. We remedy this by proposing augmentations to the basic IRLS routine that not only offer guaranteed global recovery, but in practice also outperform state-of-the-art algorithms for robust regression. …
In this contribution, we develop an efficient surrogate modeling framework for simulation-based optimization of enhanced oil recovery, where we particularly focus on polymer flooding. The computational approach is based on an adaptive training procedure of a neural network that directly approximates an input-output map of the underlying PDE-constrained optimization problem. The training process thereby focuses on the construction of an accurate surrogate model solely related to the optimization path of an outer iterative optimization loop. True evaluations of the objective function are used to…
We propose a novel, structured pruning algorithm for neural networks -- the iterative, Sparse Structured Pruning algorithm, dubbed as i-SpaSP. Inspired by ideas from sparse signal recovery, i-SpaSP operates by iteratively identifying a larger set of important parameter groups (e.g., filters or neurons) within a network that contribute most to the residual between pruned and dense network output, then thresholding these groups based on a smaller, pre-defined pruning ratio. For both two-layer and multi-layer network architectures with ReLU activations, we show the error induced by pruning with i…
