Total Variation regularization (TV) is a seminal approach for image recovery. TV involves the norm of the image's gradient, aggregated over all pixel locations. Therefore, TV leads to piece-wise constant solutions, resulting in what is known as the "staircase effect." To mitigate this effect, the Hessian Schatten norm regularization (HSN) employs second-order derivatives, represented by the pth norm of eigenvalues in the image hessian, summed across all pixels. HSN demonstrates superior structure-preserving properties compared to TV. However, HSN solutions tend to be overly smoothed. To address this, we introduce a non-convex shrinkage penalty applied to the Hessian's eigenvalues, deviating from the convex lp norm. It is important to note that the shrinkage penalty is not defined directly in closed form, but specified indirectly through its proximal operation. This makes constructing a provably convergent algorithm difficult as the singular values are also defined through a non-linear operation. However, we were able to derive a provably convergent algorithm using proximal operations. We prove the convergence by establishing that the proposed regularization adheres to restricted proximal regularity. The images recovered by this regularization were sharper than the convex counterparts.
We consider the iterative shrinkage/thresholding algorithm (ISTA) applied to a cost function composed of a data fidelity term and a penalty term. The penalty is non-convex but the concavity of the penalty is accounted for by the data fidelity term so that the overall cost function is convex. We provide a generalization of the convergence result for ISTA viewed as a forward-backward splitting algorithm. We also demonstrate experimentally that for the current setup, using large stepsizes in ISTA can accelerate convergence more than existing schemes proposed for the convex case, like TwIST or FIS…
In inverse problems, the use of an $\ell_{12}$ analysis regularizer induces a bias in the estimated solution. We propose a general refitting framework for removing this artifact while keeping information of interest contained in the biased solution. This is done through the use of refitting block penalties that only act on the co-support of the estimation. Based on an analysis of related works in the literature, we propose a new penalty that is well suited for refitting purposes. We also present an efficient algorithmic method to obtain the refitted solution along with the original (biased) so…
The constrained $\ell_0$ regularization plays an important role in sparse reconstruction. A widely used approach for solving this problem is the penalty method, of which the least square penalty problem is a special case. However, the connections between global minimizers of the constrained $\ell_0$ problem and its penalty problem have never been studied in a systematic way. This work provides a comprehensive investigation on optimal solutions of these two problems and their connections. We give detailed descriptions of optimal solutions of the two problems, including existence, stability with…
This paper develops a convex approach for sparse one-dimensional deconvolution that improves upon L1-norm regularization, the standard convex approach. We propose a sparsity-inducing non-separable non-convex bivariate penalty function for this purpose. It is designed to enable the convex formulation of ill-conditioned linear inverse problems with quadratic data fidelity terms. The new penalty overcomes limitations of separable regularization. We show how the penalty parameters should be set to ensure that the objective function is convex, and provide an explicit condition to verify the optimal…
Sparse approximate solutions to linear equations are classically obtained via L1 norm regularized least squares, but this method often underestimates the true solution. As an alternative to the L1 norm, this paper proposes a class of non-convex penalty functions that maintain the convexity of the least squares cost function to be minimized, and avoids the systematic underestimation characteristic of L1 norm regularization. The proposed penalty function is a multivariate generalization of the minimax-concave (MC) penalty. It is defined in terms of a new multivariate generalization of the Huber …
We propose a trust-region method that solves a sequence of linear integer programs to tackle integer optimal control problems regularized with a total variation penalty. The total variation penalty allows us to prove the existence of minimizers of the integer optimal control problem. We introduce a local optimality concept for the problem, which arises from the infinite-dimensional perspective. In the case of a one-dimensional domain of the control function, we prove convergence of the iterates produced by our algorithm to points that satisfy first-order stationarity conditions for local optim…
We propose a convex formulation of the fused lasso signal approximation problem consisting of non-convex penalty functions. The fused lasso signal model aims to estimate a sparse piecewise constant signal from a noisy observation. Originally, the $\ell_1$ norm was used as a sparsity-inducing convex penalty function for the fused lasso signal approximation problem. However, the $\ell_1$ norm underestimates signal values. Non-convex sparsity-inducing penalty functions better estimate signal values. In this paper, we show how to ensure the convexity of the fused lasso signal approximation problem…
In the paper, we study the minimization problem of a non-convex sparsity promoting penalty function $$P_{a}(x)=\sum_{i=1}^{n}p_{a}(x_{i})=\sum_{i=1}^{n}\frac{a|x_{i}|}{1+a|x_{i}|}$$ in compressed sensing, which is called fraction function. Firstly, we discuss the equivalence of $\ell_{0}$ minimization and fraction function minimization. It is proved that there corresponds a constant $a^{**}>0$ such that, whenever $a>a^{**}$, every solution to $(FP_{a})$ also solves $(P_{0})$, that the uniqueness of global minimizer of $(FP_{a})$ and its equivalence to $(P_{0})$ if the sensing matrix $A$ satisf…
We investigate non-convex optimization problems in $BV(Ω)$ with two-sided pointwise inequality constraints. We propose a regularization and penalization method to numerically solve the problem. Under certain conditions, weak limit points of iterates are stationary for the original problem. In addition, we prove optimality conditions for the original problem that contain Lagrange multipliers to the inequality constraints. Numerical experiments confirm the theoretical findings.
Affine matrix rank minimization problem is a fundamental problem with a lot of important applications in many fields. It is well known that this problem is combinatorial and NP-hard in general. In this paper, a continuous promoting low rank non-convex fraction function is studied to replace the rank function in this NP-hard problem. Inspired by our former work in compressed sensing, an iterative singular value thresholding algorithm is proposed to solve the regularization transformed affine matrix rank minimization problem. For different $a>0$, we could get a much better result by adjusting th…
A data filtering method for cluster analysis is proposed, based on minimizing a least squares function with a weighted $\ell_0$-norm penalty. To overcome the discontinuity of the objective function, smooth non-convex functions are employed to approximate the $\ell_0$-norm. The convergence of the global minimum points of the approximating problems towards global minimum points of the original problem is stated. The proposed method also exploits a suitable technique to choose the penalty parameter. Numerical results on synthetic and real data sets are finally provided, showing how some existing …
Global Optimality for Constrained Exploration via Penalty Regularization
Florian Wolf: , Ilyas Fatkhullin: , Niao He: 1The Computing \& Mathematical Sciences Department, California Institute of Technology, Pasadena, CA. 2Department of Computer Science, ETH Zurich, Switzerland. 3ETH AI Center, ETH Zurich, Switzerland.
The composite L0 function serves as a sparse regularizer in many applications. The algorithmic difficulty caused by the composite L0 regularization (the L0 norm composed with a linear mapping) is usually bypassed through approximating the L0 norm. We consider in this paper capped Lp approximations with $p>0$ for the composite L0 regularization problem. For each $p>0$, the capped Lp function converges to the L0 norm pointwisely as the approximation parameter tends to infinity. We point out that the capped Lp approximation problem is essentially a penalty method with an Lp penalty function for t…
Istituto Italiano di Tecnologia（意大利技术研究院）；MIT（麻省理工学院）；Università di Genova（热那亚大学）
We consider a regularized least squares problem, with regularization by structured sparsity-inducing norms, which extend the usual $\ell_1$ and the group lasso penalty, by allowing the subsets to overlap. Such regularizations lead to nonsmooth problems that are difficult to optimize, and we propose in this paper a suitable version of an accelerated proximal method to solve them. We prove convergence of a nested procedure, obtained composing an accelerated proximal method with an inner algorithm for computing the proximity operator. By exploiting the geometrical properties of the penalty, we de…
Regularization vs. Relaxation: A conic optimization perspective of statistical variable selection
Department of Mathematics, Washington State University（数学系，华盛顿州立大学）；Department of Statistics, University of Connecticut（统计系，康涅狄格大学）；Department of Industrial and Systems Engineering, University of Wisconsin-Madison（工业与系统工程系，威斯康星大学麦迪逊分校）
