We study the Bregman Augmented Lagrangian method (BALM) for solving convex problems with linear constraints. For classical Augmented Lagrangian method, the convergence rate and its relation with the proximal point method is well-understood. However, the convergence rate for BALM has not yet been thoroughly studied in the literature. In this paper, we analyze the convergence rates of BALM in terms of the primal objective as well as the feasibility violation. We also develop, for the first time, an accelerated Bregman proximal point method, that improves the convergence rate from $O(1/\sum_{k=0}^{T-1}η_k)$ to $O(1/(\sum_{k=0}^{T-1}\sqrt{η_k})^2)$, where $\{η_k\}_{k=0}^{T-1}$ is the sequence of proximal parameters. When applied to the dual of linearly constrained convex programs, this leads to the construction of an accelerated BALM, that achieves the improved rates for both primal and dual convergences.
The alternating direction method with multipliers (ADMM) has been one of most powerful and successful methods for solving various convex or nonconvex composite problems that arise in the fields of image & signal processing and machine learning. In convex settings, numerous convergence results have been established for ADMM as well as its varieties. However, due to the absence of convexity, the convergence analysis of nonconvex ADMM is generally very difficult. In this paper we study the Bregman modification of ADMM (BADMM), which includes the conventional ADMM as a special case and often leads…
In the context of augmented Lagrangian approaches for solving semidefinite programming problems, we investigate the possibility of eliminating the positive semidefinite constraint on the dual matrix by employing a factorization. Hints on how to deal with the resulting unconstrained maximization of the augmented Lagrangian are given. We further use the approximate maximum of the augmented Lagrangian with the aim of improving the convergence rate of alternating direction augmented Lagrangian frameworks. Numerical results are reported, showing the benefits of the approach.
In this paper, we study augmented Lagrangian functions for nonlinear semidefinite programming (NSDP) problems with exactness properties. The term exact is used in the sense that the penalty parameter can be taken appropriately, so a single minimization of the augmented Lagrangian recovers a solution of the original problem. This leads to reformulations of NSDP problems into unconstrained nonlinear programming ones. Here, we first establish a unified framework for constructing these exact functions, generalizing Di Pillo and Lucidi's work from 1996, that was aimed at solving nonlinear programmi…
Second-order sufficient conditions for local optimality have been playing an important role in local convergence analysis of optimization algorithms. In this paper, we demonstrate that this condition alone suffices to justify the linear convergence of the primal-dual sequence, generated by the augmented Lagrangian method for piecewise linear-quadratic composite optimization problems, even when the Lagrange multiplier in this class of problems is not unique. Furthermore, we establish the equivalence between the second-order sufficient condition and the quadratic growth condition of the augmente…
This paper establishes the iteration-complexity of an inner accelerated inexact proximal augmented Lagrangian (IAIPAL) method for solving linearly-constrained smooth nonconvex composite optimization problems that is based on the classical augmented Lagrangian (AL) function. More specifically, each IAIPAL iteration consists of inexactly solving a proximal AL subproblem by an accelerated composite gradient (ACG) method followed by a classical Lagrange multiplier update. Under the assumption that the domain of the composite function is bounded and the problem has a Slater point, it is shown that …
We study a stochastic first order primal-dual method for solving convex-concave saddle point problems over real reflexive Banach spaces using Bregman divergences and relative smoothness assumptions, in which we allow for stochastic error in the computation of gradient terms within the algorithm. We show ergodic convergence in expectation of the Lagrangian optimality gap with a rate of O(1/k) and that every almost sure weak cluster point of the ergodic sequence is a saddle point in expectation under mild assumptions. Under slightly stricter assumptions, we show almost sure weak convergence of t…
Algencan is a well established safeguarded Augmented Lagrangian algorithm introduced in [R. Andreani, E. G. Birgin, J. M. Mart\'ınez and M. L. Schuverdt, On Augmented Lagrangian methods with general lower-level constraints, SIAM Journal on Optimization 18, pp. 1286-1309, 2008]. Complexity results that report its worst-case behavior in terms of iterations and evaluations of functions and derivatives that are necessary to obtain suitable stopping criteria are presented in this work. In addition, the computational performance of a new version of the method is presented, which shows that the updat…
Augmented Lagrangian method (also called as method of multipliers) is an important and powerful optimization method for lots of smooth or nonsmooth variational problems in modern signal processing, imaging, optimal control and so on. However, one usually needs to solve the coupled and nonlinear system together and simultaneously, which is very challenging. In this paper, we proposed several semismooth Newton methods to solve the nonlinear subproblems arising in image restoration, which leads to several highly efficient and competitive algorithms for imaging processing. With the analysis of the…
In this paper, we consider the linearly constrained composite convex optimization problem, whose objective is a sum of a smooth function and a possibly nonsmooth function. We propose an inexact augmented Lagrangian (IAL) framework for solving the problem. The stopping criterion used in solving the augmented Lagrangian (AL) subproblem in the proposed IAL framework is weaker and potentially much easier to check than the one used in most of the existing IAL frameworks/methods. We analyze the global convergence and the non-ergodic convergence rate of the proposed IAL framework.
In this paper, we consider nonlinear optimization problems with nonlinear equality constraints and bound constraints on the variables. For the solution of such problems, many augmented Lagrangian methods have been defined in the literature. Here, we propose to modify one of these algorithms, namely ALGENCAN by Andreani et al., in such a way to incorporate second-order information into the augmented Lagrangian framework, using an active-set strategy. We show that the overall algorithm has the same convergence properties as ALGENCAN and an asymptotic quadratic convergence rate under suitable ass…
An Inexact Augmented Lagrangian Method for $(L_0, L_1)$-Smooth Convex Optimization
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Innopolis University（因诺波利斯大学）；Adyghe State University（阿迪格国立大学）；V. I. Vernadsky Crimean Federal University（V.I.维尔纳茨基克里米亚联邦大学）
We propose a variant of the classical augmented Lagrangian method for constrained optimization problems in Banach spaces. Our theoretical framework does not require any convexity or second-order assumptions and allows the treatment of inequality constraints with infinite-dimensional image space. Moreover, we discuss the convergence properties of our algorithm with regard to feasibility, global optimality, and KKT conditions. Some numerical results are given to illustrate the practical viability of the method.
In this paper, we propose a Robbins-Monro augmented Lagrangian method (RMALM) to solve a class of constrained stochastic convex optimization, which can be regarded as a hybrid of the Robbins-Monro type stochastic approximation method and the augmented Lagrangian method of convex optimizations. Under mild conditions, we show that the proposed algorithm exhibits a linear convergence rate. Moreover, instead of verifying a computationally intractable stopping criteria, we show that the RMALM with the increasing subproblem iteration number has a global complexity $\mathcal{O}(1/\varepsilon^{1+q})$ …
In this paper we analyze several inexact fast augmented Lagrangian methods for solving linearly constrained convex optimization problems. Mainly, our methods rely on the combination of excessive-gap-like smoothing technique developed in [15] and the newly introduced inexact oracle framework from [4]. We analyze several algorithmic instances with constant and adaptive smoothing parameters and derive total computational complexity results in terms of projections onto a simple primal set. For the basic inexact fast augmented Lagrangian algorithm we obtain the overall computational complexity of o…
This paper proposes and analyzes a proximal augmented Lagrangian (NL-IAPIAL) method for solving smooth nonconvex composite optimization problems with nonlinear $\cal K$-convex constraints, i.e., the constraints are convex with respect to the order given by a closed convex cone $\cal K$. Each NL-IAPIAL iteration consists of inexactly solving a proximal augmented Lagrangian subproblem by an accelerated composite gradient (ACG) method followed by a Lagrange multiplier update. Under some mild assumptions, it is shown that NL-IAPIAL generates an approximate stationary solution of the constrained pr…
We study the computational complexity certification of inexact gradient augmented Lagrangian methods for solving convex optimization problems with complicated constraints. We solve the augmented Lagrangian dual problem that arises from the relaxation of complicating constraints with gradient and fast gradient methods based on inexact first order information. Moreover, since the exact solution of the augmented Lagrangian primal problem is hard to compute in practice, we solve this problem up to some given inner accuracy. We derive relations between the inner and the outer accuracy of the primal…
This paper presents a new linear hyperspectral unmixing method of the minimum volume class, termed \emph{simplex identification via split augmented Lagrangian} (SISAL). Following Craig's seminal ideas, hyperspectral linear unmixing amounts to finding the minimum volume simplex containing the hyperspectral vectors. This is a nonconvex optimization problem with convex constraints. In the proposed approach, the positivity constraints, forcing the spectral vectors to belong to the convex hull of the endmember signatures, are replaced by soft constraints. The obtained problem is solved by a sequenc…
We develop a decomposition method based on the augmented Lagrangian framework to solve a broad family of semidefinite programming problems, possibly with nonlinear objective functions, nonsmooth regularization, and general linear equality/inequality constraints. In particular, the positive semidefinite variable along with a group of linear constraints can be transformed into a variable on a smooth manifold via matrix factorization. The nonsmooth regularization and other general linear constraints are handled by the augmented Lagrangian method. Therefore, each subproblem can be solved by a semi…
This paper is devoted to studying an augmented Lagrangian method for solving a class of manifold optimization problems, which have nonsmooth objective functions and nonlinear constraints. Under the constant positive linear dependence condition on manifolds, we show that the proposed method converges to a stationary point of the nonsmooth manifold optimization problem. Moreover, we propose a globalized semismooth Newton method to solve the augmented Lagrangian subproblem on manifolds efficiently. The local superlinear convergence of the manifold semismooth Newton method is also established unde…
In the second part of our study we introduce the concept of global extended exactness of penalty and augmented Lagrangian functions, and derive the localization principle in the extended form. The main idea behind the extended exactness consists in an extension of the original constrained optimization problem by adding some extra variables, and then construction of a penalty/augmented Lagrangian function for the extended problem. This approach allows one to design extended penalty/augmented Lagrangian functions having some useful properties (such as smoothness), which their counterparts for th…
Nonlinearly constrained nonconvex and nonsmooth optimization models play an increasingly important role in machine learning, statistics and data analytics. In this paper, based on the augmented Lagrangian function we introduce a flexible first-order primal-dual method, to be called nonconvex auxiliary problem principle of augmented Lagrangian (NAPP-AL), for solving a class of nonlinearly constrained nonconvex and nonsmooth optimization problems. We demonstrate that NAPP-AL converges to a stationary solution at the rate of o(1/\sqrt{k}), where k is the number of iterations. Moreover, under an a…
Mixed integer quadratic programming (MIQP) is the problem of minimizing a convex quadratic function over mixed integer points in a rational polyhedron. This paper focuses on the augmented Lagrangian dual (ALD) for MIQP. ALD augments the usual Lagrangian dual with a weighted nonlinear penalty on the dualized constraints. We first prove that ALD will reach a zero duality gap asymptotically as the weight on the penalty goes to infinity under some mild conditions on the penalty function. We next show that a finite penalty weight is enough for a zero gap when we use any norm as the penalty function…
Most recently, He and Yuan [arXiv:2108.08554, 2021] have proposed a balanced augmented Lagrangian method (ALM) for the canonical convex programming problem with linear constraints, which advances the original ALM by balancing its subproblems and improving its implementation. In this short note, we propose a dual-primal version of the balanced ALM, which updates the new iterate via a conversely dual-primal iterative order formally. The proposed method inherits all advantages of the prototype balanced ALM, and its convergence analysis can be well conducted in the context of variational inequalit…
A preconditioning strategy for the Powell-Hestenes-Rockafellar Augmented Lagrangian method (ALM) is presented. The scheme exploits the structure of the Augmented Lagrangian Hessian. It is a modular preconditioner consisting of two blocks. The first one is associated with the Lagrangian of the objective while the second administers the Jacobian of the constraints and possible low-rank corrections to the Hessian. The proposed updating strategies take advantage of ALM convergence results and avoid frequent refreshing. Constraint administration takes into account complementarity over the Lagrange …
The augmented Lagrangian method (ALM) has gained tremendous popularity for its elegant theory and impressive numerical performance since it was proposed by Hestenes and Powell in 1969. It has been widely used in numerous efficient solvers to improve numerical performance to solve many problems. In this paper, without requiring the uniqueness of multipliers, the local (asymptotic Q-superlinear) Q-linear convergence rate of the primal-dual sequences generated by ALM for the nonlinear semidefinite programming (NLSDP) is established by assuming the second-order sufficient condition (SOSC) and the …
We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hetenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality cons…
This paper proposes and analyzes an accelerated inexact dampened augmented Lagrangian (AIDAL) method for solving linearly-constrained nonconvex composite optimization problems. Each iteration of the AIDAL method consists of: (i) inexactly solving a dampened proximal augmented Lagrangian (AL) subproblem by calling an accelerated composite gradient (ACG) subroutine; (ii) applying a dampened and under-relaxed Lagrange multiplier update; and (iii) using a novel test to check whether the penalty parameter of the AL function should be increased. Under several mild assumptions involving the dampening…
This work presents an adaptive superfast proximal augmented Lagrangian (AS-PAL) method for solving linearly-constrained smooth nonconvex composite optimization problems. Each iteration of AS-PAL inexactly solves a possibly nonconvex proximal augmented Lagrangian (AL) subproblem obtained by an aggressive/adaptive choice of prox stepsize with the aim of substantially improving its computational performance followed by a full Lagrangian multiplier update. A major advantage of AS-PAL compared to other AL methods is that it requires no knowledge of parameters (e.g., size of constraint matrix, objec…
We present a numerical method for the minimization of constrained optimization problems where the objective is augmented with large quadratic penalties of inconsistent equality constraints. Such objectives arise from quadratic integral penalty methods for the direct transcription of optimal control problems. The Augmented Lagrangian Method (ALM) has a number of advantages over the Quadratic Penalty Method (QPM). However, if the equality constraints are inconsistent, then ALM might not converge to a point that minimizes the bias of the objective and penalty term. Therefore, we present a modific…
In this paper we present complexity certification results for a distributed Augmented Lagrangian (AL) algorithm used to solve convex optimization problems involving globally coupled linear constraints. Our method relies on the Accelerated Distributed Augmented Lagrangian (ADAL) algorithm, which can handle the coupled linear constraints in a distributed manner based on local estimates of the AL. We show that the theoretical complexity of ADAL to reach an $ε$-optimal solution both in terms of suboptimality and infeasibility is $O(\frac{1}ε)$ iterations. Moreover, we provide a valid upper bound f…
A novel augmented Lagrangian method for solving non-convex programs with nonlinear cost and constraint couplings in a distributed framework is presented. The proposed decomposition algorithm is made of two layers: The outer level is a standard multiplier method with penalty on the nonlinear equality constraints, while the inner level consists of a block-coordinate descent (BCD) scheme. Based on standard results on multiplier methods and recent results on proximal regularised BCD techniques, it is proven that the method converges to a KKT point of the non-convex nonlinear program under a semi-a…
We propose two basic assumptions, under which the rate of convergence of the augmented Lagrange method for a class of composite optimization problems is estimated. We analyze the rate of local convergence of the augmented Lagrangian method for a nonlinear semidefinite nuclear norm composite optimization problem by verifying these two basic assumptions. Without requiring strict complementarity, we prove that, under the constraint nondegeneracy condition and the strong second order sufficient condition, the rate of convergence is linear and the ratio constant is proportional to 1/c, where c is t…
We propose an augmented Lagrangian-type algorithm for the solution of generalized Nash equilibrium problems (GNEPs). Specifically, we discuss the convergence properties with regard to both feasibility and optimality of limit points. This is done by introducing a secondary GNEP as a new optimality concept. In this context, special consideration is given to the role of suitable constraint qualifications that take into account the particular structure of GNEPs. Furthermore, we consider the behaviour of the method for jointly-convex GNEPs and describe a modification which is tailored towards the c…
Due to the possible lack of primal-dual-type error bounds, the superlinear convergence for the Karush-Kuhn-Tucker (KKT) residues of the sequence generated by augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) has long been an outstanding open question. In this paper, we aim to resolve this issue by first conducting convergence rate analysis for the ALM with Rockafellar's stopping criteria under only a mild quadratic growth condition on the dual of CCCP. More importantly, by further assuming that the Robinson constraint qualification holds, we establish the …
The augmented Lagrangian method (ALM) is one of the most useful methods for constrained optimization. Its convergence has been well established under convexity assumptions or smoothness assumptions, or under both assumptions. ALM may experience oscillations and divergence when the underlying problem is simultaneously nonconvex and nonsmooth. In this paper, we consider the linearly constrained problem with a nonconvex (in particular, weakly convex) and nonsmooth objective. We modify ALM to use a Moreau envelope of the augmented Lagrangian and establish its convergence under conditions that are …
This paper proposes and establishes the iteration-complexity of an inexact proximal accelerated augmented Lagrangian (IPAAL) method for solving linearly constrained smooth nonconvex composite optimization problems. Each IPAAL iteration consists of inexactly solving a proximal augmented Lagrangian subproblem by an accelerated composite gradient (ACG) method followed by a suitable Lagrange multiplier update. It is shown that IPAAL generates an approximate stationary solution in at most ${\cal O}(\log(1/ρ)/ρ^{3})$ ACG iterations, where $ρ>0$ is the given tolerance. It is also shown that the previ…
We consider a misspecified optimization problem that requires minimizing of a convex function $f(x;θ^*)$ in x over a constraint set represented by $h(x;θ^*)\leq 0$, where $θ^*$ is an unknown (or misspecified) vector of parameters. Suppose $θ^*$ can be learnt by a distinct process that generates a sequence of estimators $θ_k$, each of which is an increasingly accurate approximation of $θ^*$. We develop a first-order augmented Lagrangian scheme for computing an optimal solution $x^*$ while simultaneously learning $θ^*$.
