The $\ell_1$ norm is the tight convex relaxation for the $\ell_0$ "norm" and has been successfully applied for recovering sparse signals. For problems with fewer samplings, one needs to enhance the sparsity by nonconvex penalties such as $\ell_p$ "norm". As one method for solving $\ell_p$ minimization problems, iteratively reweighted $\ell_1$ minimization updates the weight for each component based on the value of the same component at the previous iteration. It assigns large weights on small components in magnitude and small weights on large components in magnitude. In this paper, we consider a weighted $\ell_1$ penalty with the set of the weights fixed and the weights are assigned based on the sort of all the components in magnitude. The smallest weight is assigned to the largest component in magnitude. This new penalty is called nonconvex sorted $\ell_1$. Then we propose two methods for solving nonconvex sorted $\ell_1$ minimization problems: iteratively reweighted $\ell_1$ minimization and iterative sorted thresholding, and prove that both methods will converge to a local optimum. We also show that both methods are generalizations of iterative support detection and iterative hard thresholding respectively. The numerical experiments demonstrate the better performance of assigning weights by sort compared to $\ell_p$ minimization.
Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP（法国国家信息与自动化技术研究所、里昂大学、国家科学研究中心、克莱尔伯恩大学、LIP）；IRIT, Université de Toulouse, CNRS（图卢兹大学、国家科学研究中心、IRIT）
This paper presents a canonical dual approach for solving nonconvex quadratic minimization problem. By using the canonical duality theory, nonconvex primal minimization problems over n-dimensional Lorentz cone can be transformed into certain canonical dual problems with only one dual variable, which can be solved by using standard convex minimization methods. Extremality conditions of these solutions are classified by the triality theory. Applications are illustrated.
For the problem of sparse recovery, it is widely accepted that nonconvex minimizations are better than $\ell_1$ penalty in enhancing the sparsity of solution. However, to date, the theory verifying that nonconvex penalties outperform (or are at least as good as) $\ell_1$ minimization in exact, uniform recovery has mostly been limited to separable cases. In this paper, we establish general recovery guarantees through null space conditions for nonconvex, non-separable regularizations, which are slightly less demanding than the standard null space property for $\ell_1$ minimization.
By applying the perturbation function approach, we propose the Lagrangian and the conjugate duals for minimization problems of the sum of two, generally nonconvex, functions. The main tools are the $Φ$-convexity theory and minimax theorems for $Φ$-convex functions. We provide conditions ensuring zero duality gap and introduce $Φ$-Karush-Kuhn-Tucker conditions that characterize solutions to primal and dual problems. We also discuss the relationship between the dual problems introduced in the present investigation and some conjugate-type duals existing in the literature.
An adaptive regularization algorithm using inexact function and derivatives evaluations is proposed for the solution of composite nonsmooth nonconvex optimization. It is shown that this algorithm needs at most $O(|\log(ε)|\,ε^{-2})$ evaluations of the problem's functions and their derivatives for finding an $ε$-approximate first-order stationary point. This complexity bound therefore generalizes that provided by [Bellavia, Gurioli, Morini and Toint, 2018] for inexact methods for smooth nonconvex problems, and is within a factor $|\log(ε)|$ of the optimal bound known for smooth and nonsmooth no…
We extend the Malitsky-Tam forward-reflected-backward (FRB) splitting method for inclusion problems of monotone operators to nonconvex minimization problems. By assuming the generalized concave Kurdyka-Łojasiewicz (KL) property of a quadratic regularization of the objective, we show that the FRB method converges globally to a stationary point of the objective and enjoys finite length property. The sharpness of our approach is guaranteed by virtue of the exact modulus associated with the generalized concave KL property. Numerical experiments suggest that FRB is competitive compared to the Dougl…
In this paper we study nonconvex and nonsmooth optimization problems with semi-algebraic data, where the variables vector is split into several blocks of variables. The problem consists of one smooth function of the entire variables vector and the sum of nonsmooth functions for each block separately. We analyze an inertial version of the Proximal Alternating Linearized Minimization (PALM) algorithm and prove its global convergence to a critical point of the objective function at hand. We illustrate our theoretical findings by presenting numerical experiments on blind image deconvolution, on sp…
This paper studies a structured compound stochastic program (SP) involving multiple expectations coupled by nonconvex and nonsmooth functions. We present a successive convex-programming based sampling algorithm and establish its subsequential convergence. We describe stationarity properties of the limit points for several classes of the compound SP. We further discuss probabilistic stopping rules based on the computable error-bound for the algorithm. We present several risk measure minimization problems that can be formulated as such a compound stochastic program; these include generalized dev…
Minimization of a smooth function on a sphere or, more generally, on a smooth manifold, is the simplest non-convex optimization problem. It has a lot of applications. Our goal is to propose a version of the gradient projection algorithm for its solution and to obtain results that guarantee convergence of the algorithm under some minimal natural assumptions. We use the Lezanski-Polyak-Lojasiewicz condition on a manifold to prove the global linear convergence of the algorithm. Another method well fitted for the problem is the conditional gradient (Frank-Wolfe) algorithm. We examine some conditio…
We consider minimization of indefinite quadratics with either trust-region (norm) constraints or cubic regularization. Despite the nonconvexity of these problems we prove that, under mild assumptions, gradient descent converges to their global solutions, and give a non-asymptotic rate of convergence for the cubic variant. We also consider Krylov subspace solutions and establish sharp convergence guarantees to the solutions of both trust-region and cubic-regularized problems. Our rates mirror the behavior of these methods on convex quadratics and eigenvector problems, highlighting their scalabi…
Variable Bregman Majorization-Minimization algorithms for nonconvex nonsmooth optimization, with application to Poisson imaging
Université Paris-Saclay, CentraleSupélec, Inria, Centre pour la Vision Numérique（巴黎萨克雷大学、中央苏佩莱克学院、法国国家信息与自动化研究所、数字视觉中心）；Université Paris-Saclay, Laboratoire BioMaps, CEA（巴黎萨克雷大学、BioMaps实验室、法国原子能和替代能源委员会）
We consider a class of nonsmooth and nonconvex optimization problems over the Stiefel manifold where the objective function is the summation of a nonconvex smooth function and a nonsmooth Lipschitz continuous convex function composed with an linear mapping. We propose three numerical algorithms for solving this problem, by combining smoothing methods and some existing algorithms for smooth optimization over the Stiefel manifold. In particular, we approximate the aforementioned nonsmooth convex function by its Moreau envelope in our smoothing methods, and prove that the Moreau envelope has many…
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
We propose a Bregman inertial forward-reflected-backward (BiFRB) method for nonconvex composite problems. Our analysis relies on a novel approach that imposes general conditions on implicit merit function parameters, which yields a stepsize condition that is independent of inertial parameters. In turn, a question of Malitsky and Tam regarding whether FRB can be equipped with a Nesterov-type acceleration is resolved. Assuming the generalized concave Kurdyka-Łojasiewicz property of a quadratic regularization of the objective, we obtain sequential convergence of BiFRB, as well as convergence rate…
We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function, each of the nonsmooth summands depending on an independent block variable, and a smooth function which couples the two block variables. The algorithm is a full splitting method, which means that the nonsmooth functions are processed via their proximal operators, the smooth function via gradient steps, and the linear operator via matrix times vector multiplication. We provide sufficient conditions for the bou…
The affine matrix rank minimization (AMRM) problem is to find a matrix of minimum rank that satisfies a given linear system constraint. It has many applications in some important areas such as control, recommender systems, matrix completion and network localization. However, the problem (AMRM) is NP-hard in general due to the combinational nature of the matrix rank function. There are many alternative functions have been proposed to substitute the matrix rank function, which lead to many corresponding alternative minimization problems solved efficiently by some popular convex or nonconvex opti…
Detecting hidden convexity is one of the tools to address nonconvex minimization problems. After giving a formal definition of hidden convexity, we introduce the notion of conditional infimum, as it will prove instrumental in detecting hidden convexity. We develop the theory of the conditional infimum, and we establish a tower property, relevant for minimization problems. Thus equipped, we provide a sufficient condition for hidden convexity in nonconvex minimization problems. We illustrate our result on nonconvex quadratic minimization problems. We conclude with perspectives for using the cond…
In this paper, we consider an accelerated method for solving nonconvex and nonsmooth minimization problems. We propose a Bregman Proximal Gradient algorithm with extrapolation(BPGe). This algorithm extends and accelerates the Bregman Proximal Gradient algorithm (BPG), which circumvents the restrictive global Lipschitz gradient continuity assumption needed in Proximal Gradient algorithms (PG). The BPGe algorithm has higher generality than the recently introduced Proximal Gradient algorithm with extrapolation(PGe), and besides, due to the extrapolation step, BPGe converges faster than BPG algori…
A tensor nuclear norm (TNN) based method for solving the tensor recovery problem was recently proposed, and it has achieved state-of-the-art performance. However, it may fail to produce a highly accurate solution since it tends to treats each frontal slice and each rank component of each frontal slice equally. In order to get a recovery with high accuracy, we propose a general and flexible rank relaxation function named double weighted nonconvex nonsmooth rank (DWNNR) relaxation function for efficiently solving the third order tensor recovery problem. The DWNNR relaxation function can be deriv…
As a natural extension of compressive sensing and the requirement of some practical problems, Phaseless Compressed Sensing (PCS) has been introduced and studied recently. Many theoretical results have been obtained for PCS with the aid of its convex relaxation. Motivated by successful applications of nonconvex relaxed methods for solving compressive sensing, in this paper, we try to investigate PCS via its nonconvex relaxation. Specifically, we relax PCS in the real context by the corresponding $\ell_p$-minimization with $p\in (0,1)$. We show that there exists a constant $p^\ast\in (0,1]$ such…
Error bound analysis, which estimates the distance of a point to the solution set of an optimization problem using the optimality residual, is a powerful tool for the analysis of first-order optimization algorithms. In this paper, we use global error bound analysis to study the iteration complexity of a first-order algorithm for a linearly constrained nonconvex minimization problem. we develop a global dual error bound analysis for a regularized version of this nonconvex problem by using a novel ``decomposition'' technique. Equipped with this global dual error bound, we prove that a suitably d…
We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function satisfies the Kurdyka-Łojasiewicz property. We also provide convergence rates for the trajectory formulated in terms of the Łojasiewicz exponent.
We study a canonical duality method to solve a mixed-integer nonconvex fourth-order polynomial minimization problem with fixed cost terms. This constrained nonconvex problem can be transformed into a continuous concave maximization dual problem without duality gap. The global optimality conditions are proposed and the existence and uniqueness criteria are discussed. Application to a decoupled mixed-integer problem is illustrated and analytic solution for a global minimum is obtained under some suitable conditions. Several examples are given to show the method is effective.
Rank minimization is of interest in machine learning applications such as recommender systems and robust principal component analysis. Minimizing the convex relaxation to the rank minimization problem, the nuclear norm, is an effective technique to solve the problem with strong performance guarantees. However, nonconvex relaxations have less estimation bias than the nuclear norm and can more accurately reduce the effect of noise on the measurements. We develop efficient algorithms based on iteratively reweighted nuclear norm schemes, while also utilizing the low rank factorization for semidefi…
Consider the minimization of a nonconvex differentiable function over a polyhedron. A popular primal-dual first-order method for this problem is to perform a gradient projection iteration for the augmented Lagrangian function and then update the dual multiplier vector using the constraint residual. However, numerical examples show that this approach can exhibit "oscillation" and may not converge. In this paper, we propose a proximal alternating direction method of multipliers for the multi-block version of this problem. A distinctive feature of this method is the introduction of a "smoothed" (…
We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: $L(x,y)=f(x)+Q(x,y)+g(y)$, where $f:\R^n\rightarrow\R\cup{+\infty}$ and $g:\R^m\rightarrow\R\cup{+\infty}$ are proper lower semicontinuous functions, and $Q:\R^n\times\R^m\rightarrow \R$ is a smooth $C^1$ function which couples the variables $x$ and $y$. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize $L$. We work in a nonconvex setting, just assuming that the function $L$ satisfies the Kurdyka-Łojasiewicz in…
We address the minimization of the sum of a proper, convex and lower semicontinuous with a (possibly nonconvex) smooth function from the perspective of an implicit dynamical system of forward-backward type. The latter is formulated by means of the gradient of the smooth function and of the proximal point operator of the nonsmooth one. The trajectory generated by the dynamical system is proved to asymptotically converge to a critical point of the objective, provided a regularization of the latter satisfies the Kurdyka-Łojasiewicz property. Convergence rates for the trajectory in terms of the Ło…
We analyze the performance of alternating minimization for loss functions optimized over two variables, where each variable may be restricted to lie in some potentially nonconvex constraint set. This type of setting arises naturally in high-dimensional statistics and signal processing, where the variables often reflect different structures or components within the signals being considered. Our analysis relies on the notion of local concavity coefficients, which has been proposed in Barber and Ha to measure and quantify the concavity of a general nonconvex set. Our results further reveal import…
Regularized coordinate minimization for nonconvex composite optimization with application to quantized image compression
Politehnica Bucharest（布加勒斯特理工大学）；Gheorghe Mihoc-Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy（罗马尼亚科学院格奥尔基·米霍克-卡尤斯·雅各布数学统计与应用数学研究所）
For some typical and widely used non-convex half-quadratic regularization models and the Ambrosio-Tortorelli approximate Mumford-Shah model, based on the Kurdyka-Łojasiewicz analysis and the recent nonconvex proximal algorithms, we developed an efficient preconditioned framework aiming at the linear subproblems that appeared in the nonlinear alternating minimization procedure. Solving large-scale linear subproblems is always important and challenging for lots of alternating minimization algorithms. By cooperating the efficient and classical preconditioned iterations into the nonlinear and nonc…
We formulate bang-bang, purification, and minimization principles in dual Banach spaces with Gelfand integrals and provide a complete characterization of the saturation property of finite measure spaces. We also present a new application of the relaxation technique to large economies with infinite-dimensional commodity spaces, where the space of agents is modeled as a finite measure space. We propose a "relaxation" of large economies, which is regarded as a reasonable convexification of original economies. Under the saturation hypothesis, the relaxation and purification techniques enable us to…
Many problems in high-dimensional statistics and optimization involve minimization over nonconvex constraints-for instance, a rank constraint for a matrix estimation problem-but little is known about the theoretical properties of such optimization problems for a general nonconvex constraint set. In this paper we study the interplay between the geometric properties of the constraint set and the convergence behavior of gradient descent for minimization over this set. We develop the notion of local concavity coefficients of the constraint set, measuring the extent to which convexity is violated, …
Automated methods for facial animation are a necessary tool in the modern industry since the standard blendshape head models consist of hundreds of controllers and a manual approach is painfully slow. Different solutions have been proposed that produce output in real-time or generalize well for different face topologies. However, all these prior works consider a linear approximation of the blendshape function and hence do not provide a high-enough level of details for modern realistic human face reconstruction. We build a method for solving the inverse rig in blendshape animation using quadrat…
Motivated by variational models in continuum mechanics, we introduce a novel algorithm to perform nonsmooth and nonconvex minimizations with linear constraints in Euclidean spaces. We show how this algorithm is actually a natural generalization of the well-known non-stationary augmented Lagrangian method for convex optimization. The relevant features of this approach are its applicability to a large variety of nonsmooth and nonconvex objective functions, its guaranteed convergence to critical points of the objective energy independently of the choice of the initial value, and its simplicity of…
Difference-of-Convex (DC) minimization, referring to the problem of minimizing the difference of two convex functions, has been found rich applications in statistical learning and studied extensively for decades. However, existing methods are primarily based on multi-stage convex relaxation, only leading to weak optimality of critical points. This paper proposes a coordinate descent method for minimizing a class of DC functions based on sequential nonconvex approximation. Our approach iteratively solves a nonconvex one-dimensional subproblem globally, and it is guaranteed to converge to a coor…
