Plug-and-Play (PnP) methods solve ill-posed inverse problems through iterative proximal algorithms by replacing a proximal operator by a denoising operation. When applied with deep neural network denoisers, these methods have shown state-of-the-art visual performance for image restoration problems. However, their theoretical convergence analysis is still incomplete. Most of the existing convergence results consider nonexpansive denoisers, which is non-realistic, or limit their analysis to strongly convex data-fidelity terms in the inverse problem to solve. Recently, it was proposed to train the denoiser as a gradient descent step on a functional parameterized by a deep neural network. Using such a denoiser guarantees the convergence of the PnP version of the Half-Quadratic-Splitting (PnP-HQS) iterative algorithm. In this paper, we show that this gradient denoiser can actually correspond to the proximal operator of another scalar function. Given this new result, we exploit the convergence theory of proximal algorithms in the nonconvex setting to obtain convergence results for PnP-PGD (Proximal Gradient Descent) and PnP-ADMM (Alternating Direction Method of Multipliers). When built on top of a smooth gradient denoiser, we show that PnP-PGD and PnP-ADMM are convergent and target stationary points of an explicit functional. These convergence results are confirmed with numerical experiments on deblurring, super-resolution and inpainting.
Plug-and-Play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. Although Plug-and-Play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. In this work, we propose a new type of Plug-and-Play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep…
In most state-of-the-art image restoration methods, the sum of a data-fidelity and a regularization term is optimized using an iterative algorithm such as ADMM (alternating direction method of multipliers). In recent years, the possibility of using denoisers for regularization has been explored in several works. A popular approach is to formally replace the proximal operator within the ADMM framework with some powerful denoiser. However, since most state-of-the-art denoisers cannot be posed as a proximal operator, one cannot guarantee the convergence of these so-called plug-and-play (PnP) algo…
The proximal gradient algorithm has been popularly used for convex optimization. Recently, it has also been extended for nonconvex problems, and the current state-of-the-art is the nonmonotone accelerated proximal gradient algorithm. However, it typically requires two exact proximal steps in each iteration, and can be inefficient when the proximal step is expensive. In this paper, we propose an efficient proximal gradient algorithm that requires only one inexact (and thus less expensive) proximal step in each iteration. Convergence to a critical point %of the nonconvex problem is still guarant…
In this paper we present the proximal point method for a special class of nonconvex function on a Hadamard manifold. The well definedness of the sequence generated by the proximal point method is guaranteed. Moreover, it is proved that each accumulation point of this sequence satisfies the necessary optimality conditions and, under additional assumptions, its convergence for a minimizer is obtained.
The proximal point method for a special class of nonconvex multiobjective functions is studied in this paper. We show that the method is well defined and that the accumulation points of any generated sequence, if any, are Pareto--Clarke critical points. Moreover, under additional assumptions, we show the full convergence of the generated sequence.
Local convergence analysis of the proximal point method for special class of nonconvex function on Hadamard manifold is presented in this paper. The well definedness of the sequence generated by the proximal point method is guaranteed. Moreover, is proved that each cluster point of this sequence satisfies the necessary optimality conditions and, under additional assumptions, its convergence for a minimizer is obtained.
Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated gradient method to solve a nonconvex stochastic composite optimization problem where the objective is the sum of smooth and nonsmooth functions, the constraint functions are assumed to be deterministic and the solution to the proximal map of the nonsmooth part is calculated inexactly at each iteration. We demonstrate an asymptotic sublinear rate of convergence for…
We investigate the convergence of an application of a proximal gradient method to control problems with nonsmooth and nonconvex control cost. Here, we focus on control cost functionals that promote sparsity, which includes functionals of $L^p$-type for $p\in [0,1)$. We prove stationarity properties of weak limit points of the method. These properties are weaker than those provided by Pontryagin's maximum principle and weaker than $L$-stationarity.
This paper establishes convergence rate bounds for a variant of the proximal alternating direction method of multipliers (ADMM) for solving nonconvex linearly constrained optimization problems. The variant of the proximal ADMM allows the inclusion of an over-relaxation stepsize parameter belonging to the interval $(0,2)$. To the best of our knowledge, all related papers in the literature only consider the case where the over-relaxation parameter lies in the interval $(0,(1+\sqrt{5})/2)$.
This paper describes and establishes the iteration-complexity of a doubly accelerated inexact proximal point (D-AIPP) method for solving the nonconvex composite minimization problem whose objective function is of the form $f+h$ where $f$ is a (possibly nonconvex) differentiable function whose gradient is Lipschitz continuous and $h$ is a closed convex function with bounded domain. D-AIPP performs two types of iterations, namely, inner and outer ones. Its outer iterations correspond to the ones of the accelerated inexact proximal point scheme. Its inner iterations are the ones performed by an a…
In this paper, we consider a proximal linearized alternating direction method of multipliers (PL-ADMM) for solving linearly constrained nonconvex and possibly nonsmooth optimization problems. The algorithm is generalized by using variable metric proximal terms in the primal updates and an over-relaxation stepsize in the multiplier update. We prove that the sequence generated by this method is bounded and its limit points are critical points. Under the powerful Kurdyka-{Łojasiewicz} properties we prove that the sequence has a finite length thus converges, and we drive its convergence rates.
A standard model for image reconstruction involves the minimization of a data-fidelity term along with a regularizer, where the optimization is performed using proximal algorithms such as ISTA and ADMM. In plug-and-play (PnP) regularization, the proximal operator (associated with the regularizer) in ISTA and ADMM is replaced by a powerful image denoiser. Although PnP regularization works surprisingly well in practice, its theoretical convergence -- whether convergence of the PnP iterates is guaranteed and if they minimize some objective function -- is not completely understood even for simple …
In this paper, we study the proximal incremental aggregated gradient(PIAG) algorithm for minimizing the sum of L-smooth nonconvex component functions and a proper closed convex function. By exploiting the L-smooth property and with the help of an error bound condition, we can show that the PIAG method still enjoys some nice linear convergence properties even for nonconvex minimization. To illustrate this, we first demonstrate that the generated sequence globally converges to the stationary point set. Then, there exists a threshold such that the objective function value sequence and the iterate…
It is well-known that the lower bound of iteration complexity for solving nonconvex unconstrained optimization problems is $Ω(1/ε^2)$, which can be achieved by standard gradient descent algorithm when the objective function is smooth. This lower bound still holds for nonconvex constrained problems, while it is still unknown whether a first-order method can achieve this lower bound. In this paper, we show that a simple single-loop first-order algorithm called smoothed proximal augmented Lagrangian method (ALM) can achieve such iteration complexity lower bound. The key technical contribution is …
We tackle highly nonconvex, nonsmooth composite optimization problems whose objectives comprise a Moreau-Yosida regularized term. Classical nonconvex proximal splitting algorithms, such as nonconvex ADMM, suffer from lack of convergence for such a problem class. To overcome this difficulty, in this work we consider a lifted variant of the Moreau-Yosida regularized model and propose a novel multiblock primal-dual algorithm that intrinsically stabilizes the dual block. We provide a complete convergence analysis of our algorithm and identify respective optimality qualifications under which statio…
This paper presents smoothing schemes for obtaining approximate stationary points of unconstrained or linearly-constrained composite nonconvex-concave min-max (and hence nonsmooth) problems by applying well-known algorithms to composite smooth approximations of the original problems. More specifically, in the unconstrained (resp. constrained) case, approximate stationary points of the original problem are obtained by applying, to its composite smooth approximation, an accelerated inexact proximal point (resp. quadratic penalty) method presented in a previous paper by the authors. Iteration com…
In this paper we study the problems of minimizing the sum of two nonconvex functions: one is differentiable and satisfies smooth adaptable property. The smooth adaptable property, also named relatively smooth condition, is weaker than the globally gradient Lipschitz continuity. We analyze an inertial version of the Bregman Proximal Gradient (BPG) algorithm and prove its stationary convergence. Besides, we prove a sublinear convergence of the inertial algorithm. Moreover, if the objective function satisfies Kurdyka--Łojasiewicz (KL) property, its global convergence to a critical point of the ob…
Optimization problems with composite functions consist of an objective function which is the sum of a smooth and a (convex) nonsmooth term. This particular structure is exploited by the class of proximal gradient methods and some of their generalizations like proximal Newton and quasi-Newton methods. In this paper, we propose a regularized proximal quasi-Newton method whose main features are: (a) the method is globally convergent to stationary points, (b) the globalization is controlled by a regularization parameter, no line search is required, (c) the method can be implemented very efficientl…
In this paper we study nonconvex and nonsmooth optimization problems with semi-algebraic data, where the variables vector is split into several blocks of variables. The problem consists of one smooth function of the entire variables vector and the sum of nonsmooth functions for each block separately. We analyze an inertial version of the Proximal Alternating Linearized Minimization (PALM) algorithm and prove its global convergence to a critical point of the objective function at hand. We illustrate our theoretical findings by presenting numerical experiments on blind image deconvolution, on sp…
Unlike convex case, a local equilibrium point of a nonconvex Nash-Cournot oligopolistic equilibrium problem may not be a global one. Finding such a local equilibrium point or even a stationary point of this problem is not an easy task. This paper deals with a numerical method for Nash-Cournot equilibrium models involving nonconvex cost functions. We develop a local method to compute a stationary point of this class of problems. The convergence of the algorithm is proved and its complexity is estimated under certain assumptions. Numerical examples are implemented to illustrate the convergence b…
This work presents an adaptive superfast proximal augmented Lagrangian (AS-PAL) method for solving linearly-constrained smooth nonconvex composite optimization problems. Each iteration of AS-PAL inexactly solves a possibly nonconvex proximal augmented Lagrangian (AL) subproblem obtained by an aggressive/adaptive choice of prox stepsize with the aim of substantially improving its computational performance followed by a full Lagrangian multiplier update. A major advantage of AS-PAL compared to other AL methods is that it requires no knowledge of parameters (e.g., size of constraint matrix, objec…
This paper analyzes the iteration-complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. More specifically, the objective function is of the form $f + h$ where $f$ is a differentiable function whose gradient is Lipschitz continuous and $h$ is a closed convex function with bounded domain. The method, basically, consists of applying an accelerated inexact proximal point method for solving approximately a sequence of quadratic penalized subproblems associated to the linearly constrained problem. Each subproblem of…
For finite-dimensional problems, stochastic approximation methods have long been used to solve stochastic optimization problems. Their application to infinite-dimensional problems is less understood, particularly for nonconvex objectives. This paper presents convergence results for the stochastic proximal gradient method applied to Hilbert spaces, motivated by optimization problems with partial differential equation (PDE) constraints with random inputs and coefficients. We study stochastic algorithms for nonconvex and nonsmooth problems, where the nonsmooth part is convex and the nonconvex par…
This paper proposes and analyzes a proximal augmented Lagrangian (NL-IAPIAL) method for solving smooth nonconvex composite optimization problems with nonlinear $\cal K$-convex constraints, i.e., the constraints are convex with respect to the order given by a closed convex cone $\cal K$. Each NL-IAPIAL iteration consists of inexactly solving a proximal augmented Lagrangian subproblem by an accelerated composite gradient (ACG) method followed by a Lagrange multiplier update. Under some mild assumptions, it is shown that NL-IAPIAL generates an approximate stationary solution of the constrained pr…
Within the unmanageably large class of nonconvex optimization, we consider the rich subclass of nonsmooth problems that have composite objectives---this already includes the extensively studied convex, composite objective problems as a special case. For this subclass, we introduce a powerful, new framework that permits asymptotically non-vanishing perturbations. In particular, we develop perturbation-based batch and incremental (online like) nonconvex proximal splitting algorithms. To our knowledge, this is the first time that such perturbation-based nonconvex splitting algorithms are being pr…
We analyze worst-case complexity of a Proximal augmented Lagrangian (Proximal AL) framework for nonconvex optimization with nonlinear equality constraints. When an approximate first-order (second-order) optimal point is obtained in the subproblem, an $ε$ first-order (second-order) optimal point for the original problem can be guaranteed within $\mathcal{O}(1/ ε^{2 - η})$ outer iterations (where $η$ is a user-defined parameter with $η\in[0,2]$ for the first-order result and $η\in [1,2]$ for the second-order result) when the proximal term coefficient $β$ and penalty parameter $ρ$ satisfy $β= \ma…
Inria, ENS de Lyon, CNRS, Université Claude Bernard Lyon 1, LIP（法国国家信息与自动化技术研究所、里昂大学、国家科学研究中心、克莱尔伯恩大学、LIP）；IRIT, Université de Toulouse, CNRS（图卢兹大学、国家科学研究中心、IRIT）
We develop a new variational approach on level sets aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. With this new approach, we are able to extend the concepts of Bregman proximal mapping and their corresponding Bregman proximal envelops, Bregman proximal gap function to nonconvex setting. Properties of these mappings and functions are examined. An aim of this work is to provide a solid foundation on which further design and analysis of VBPG for more general nonconvex optimization …
This paper proposes an efficient adaptive variant of a quadratic penalty accelerated inexact proximal point (QP-AIPP) method proposed earlier by the authors. Both the QP-AIPP method and its variant solve linearly set constrained nonconvex composite optimization problems using a quadratic penalty approach where the generated penalized subproblems are solved by a variant of the underlying AIPP method. The variant, in turn, solves a given penalized subproblem by generating a sequence of proximal subproblems which are then solved by an accelerated composite gradient algorithm. The main difference …
This paper proposes and analyzes a dampened proximal alternating direction method of multipliers (DP.ADMM) for solving linearly-constrained nonconvex optimization problems where the smooth part of the objective function is nonseparable. Each iteration of DP.ADMM consists of: (i) a sequence of partial proximal augmented Lagrangian (AL) updates, (ii) an under-relaxed Lagrange multiplier update, and (iii) a novel test to check whether the penalty parameter of the AL function should be updated. Under a basic Slater condition and some requirements related to the dampening factor and under-relaxatio…
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
We first propose a decentralized proximal stochastic gradient tracking method (DProxSGT) for nonconvex stochastic composite problems, with data heterogeneously distributed on multiple workers in a decentralized connected network. To save communication cost, we then extend DProxSGT to a compressed method by compressing the communicated information. Both methods need only $\mathcal{O}(1)$ samples per worker for each proximal update, which is important to achieve good generalization performance on training deep neural networks. With a smoothness condition on the expected loss function (but not on…
Zhejiang University City College（浙大城市学院）；Southwest Jiaotong University（西南交通大学）；Zhejiang University（浙江大学）
In this paper, we introduce a proximal-proximal majorization-minimization (PPMM) algorithm for nonconvex tuning-free robust regression problems. The basic idea is to apply the proximal majorization-minimization algorithm to solve the nonconvex problem with the inner subproblems solved by a sparse semismooth Newton (SSN) method based proximal point algorithm (PPA). We must emphasize that the main difficulty in the design of the algorithm lies in how to overcome the singular difficulty of the inner subproblem. Furthermore, we also prove that the PPMM algorithm converges to a d-stationary point. …
We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function satisfies the Kurdyka-Łojasiewicz property. We also provide convergence rates for the trajectory formulated in terms of the Łojasiewicz exponent.
