We develop a Lagrange multiplier theory for nonconvex set-valued optimization problems under Lipschitz-type regularity conditions. Instead of classical continuous linear functionals, we introduce closed convex processes -- set-valued mappings whose graphs are closed convex cones -- as generalized Lagrange multipliers. This geometric framework extends separation principles beyond convexity and differentiability. We establish the existence of multiplier processes under verifiable assumptions, including Lipschitz regularity at a reference point, the existence of a bounded base of the ordering cone, and a nondegeneracy condition ensuring proper isolation of optimal values. These processes preserve global optimality: nondominated (respectively, minimal) solutions of the primal problem remain nondominated (respectively, minimal) in the penalized problem. In the scalar case, we obtain a one-to-one correspondence between multiplier processes and lower semicontinuous sublinear functions, yielding exact penalty formulations without additional constraint qualifications. An infinite-dimensional example shows that interiority conditions on the ordering cone, while sufficient, are not necessary. Applications to set-valued vector equilibrium problems are also discussed.
Based on a characterization of the optimality of a feasible solution of a convex entropy minimization problem, one shows that the feasible solutions obtained using formally the Lagrange multipliers method are optimal.
In this paper, we establish a general result on spherical maxima sharing the same Lagrange multiplier of which the following is a particular consequence: Let $X$ be a real Hilbert space. For each $r>0$, let $S_r=\{x\in X : \|x\|^2=r\}$. Let $J:X\to {\bf R}$ be a sequentially weakly upper semicontinuous functional which is Gâteaux differentiable in $X\setminus \{0\}$. Assume that $$\limsup_{x\to 0}{{J(x)}\over {\|x\|^2}}=+\infty\ .$$ Then, for each $ρ>0$, there exists an open interval $I\subseteq ]0,+\infty[$ and an increasing function $φ:I\to ]0,ρ[$ such that, for each $λ\in I$, one has $$\emp…
The concept of uniform tangent sets was introduced and discussed in [3 - Krastanov, Ribarska, SIAM J. Control Optim., 55(3), 2017]. This study is devoted to their further investigation and to generalization of the abstract Lagrange multiplier rule from [3].
In this note, three Lagrange multiplier rules introduced in the literature for set valued optimization problems are compared. A generalization of all three results is given which proves that under rather mild assumptions, $x$ is a weak solution to the constrained problem, if and only if it is a weak solution to the Lagrangian.
In this work we provide a first order sensitivity analysis of some parameterized stochastic optimal control problems. The parameters can be given by random processes. The main tool is the one-to-one correspondence between the adjoint states appearing in a weak form of the stochastic Pontryagin principle and the Lagrange multipliers associated to the state equation.
This work investigates the theoretical performance of the alternating-direction method of multipliers (ADMM) as it applies to nonconvex optimization problems, and in particular, problems with nonconvex constraint sets. The alternating direction method of multipliers is an optimization method that has largely been analyzed for convex problems. The ultimate goal is to assess what kind of theoretical convergence properties the method has in the nonconvex case, and to this end, theoretical contributions are two-fold. First, this work analyzes the method with local solution of the ADMM subproblems,…
We use an extension to the infinite dimension of the rank theorem of the differential calculus to establish a Karush-Huhn-Tucker theorem for optimization problems in Banach spaces. We provide an application to variational problems on bounded processus under equality constraints.
This paper addresses a class of constrained optimization problems over networks in which local cost functions and constraints can be nonconvex. We propose an asynchronous distributed optimization algorithm, relying on the centralized Method of Multipliers, in which each node wakes up in an uncoordinated fashion and performs either a descent step on a local Augmented Lagrangian or an ascent step on the local multiplier vector. These two phases are regulated by a distributed logic-AND, which allows nodes to understand when the descent on the (whole) Augmented Lagrangian is sufficiently small. We…
We analyze sequences generated by interior point methods (IPMs) in convex and nonconvex settings. We prove that moving the primal feasibility at the same rate as the barrier parameter $μ$ ensures the Lagrange multiplier sequence remains bounded, provided the limit point of the primal sequence has a Lagrange multiplier. This result does not require constraint qualifications. We also guarantee the IPM finds a solution satisfying strict complementarity if one exists. On the other hand, if the primal feasibility is reduced too slowly, then the algorithm converges to a point of minimal complementar…
This paper proposes tight semidefinite relaxations for polynomial optimization. The optimality conditions are investigated. We show that generally Lagrange multipliers can be expressed as polynomial functions in decision variables over the set of critical points. The polynomial expressions can be determined by linear equations. Based on these expressions, new Lasserre type semidefinite relaxations are constructed for solving polynomial optimization. We show that the hierarchy of new relaxations has finite convergence, or equivalently, the new relaxations are tight for a finite relaxation order…
We discuss first order optimality conditions for geometric optimization problems with Neumann boundary conditions and boundary observation. The methods we develop here are applicable to large classes of state systems or cost functionals. Our approach is based on the implicit parametrization theorem and the use of Hamiltonian systems. It establishes equivalence with a constrained optimal control problem and uses Lagrange multipliers under a new simple constraint qualification. In this setting, general functional variations are performed, that combine topological and boundary variations in a nat…
The alternating direction method with multipliers (ADMM) has been one of most powerful and successful methods for solving various convex or nonconvex composite problems that arise in the fields of image & signal processing and machine learning. In convex settings, numerous convergence results have been established for ADMM as well as its varieties. However, due to the absence of convexity, the convergence analysis of nonconvex ADMM is generally very difficult. In this paper we study the Bregman modification of ADMM (BADMM), which includes the conventional ADMM as a special case and often leads…
In this paper we apply an augmented Lagrange method to a class of semilinear elliptic optimal control problems with pointwise state constraints. We show strong convergence of subsequences of the primal variables to a local solution of the original problem as well as weak convergence of the adjoint states and weak* convergence of the multipliers associated to the state constraint. Moreover, we show existence of stationary points in arbitrary small neighborhoods of local solutions of the original problem. Additionally, various numerical results are presented.
The alternating direction method of multipliers (ADMM) is widely used to solve large-scale linearly constrained optimization problems, convex or nonconvex, in many engineering fields. However there is a general lack of theoretical understanding of the algorithm when the objective function is nonconvex. In this paper we analyze the convergence of the ADMM for solving certain nonconvex consensus and sharing problems, and show that the classical ADMM converges to the set of stationary solutions, provided that the penalty parameter in the augmented Lagrangian is chosen to be sufficiently large. Fo…
The paper concerns the study of criticality of Lagrange multipliers in variational systems that has been recognized in both theoretical and numerical aspects of optimization and variational analysis. In contrast to the previous developments dealing with polyhedral KKT systems and the like, we now focus on general nonpolyhedral systems that are associated, in particular, with problems of conic programming. Developing a novel approach, which is mainly based on advanced techniques and tools of second-order variational analysis and generalized differentiation, allows us to overcome principal chall…
This paper studies bilevel polynomial optimization problems. To solve them, we give a method based on polynomial optimization relaxations. Each relaxation is obtained from the Kurash-Kuhn-Tucker (KKT) conditions for the lower level optimization and the exchange technique for semi-infinite programming. For KKT conditions, Lagrange multipliers are represented as polynomial or rational functions. The Moment-SOS relaxations are used to solve the polynomial optimizattion relaxations. Under some general assumptions, we prove the convergence of the algorithm for solving bilevel polynomial optimizatio…
The classic Alternating Direction Method of Multipliers (ADMM) is a popular framework to solve linear-equality constrained problems. In this paper, we extend the ADMM naturally to nonlinear equality-constrained problems, called neADMM. The difficulty of neADMM is to solve nonconvex subproblems. We provide globally optimal solutions to them in two important applications. Experiments on synthetic and real-world datasets demonstrate excellent performance and scalability of our proposed neADMM over existing state-of-the-start methods.
The paper is devoted to the study and applications of criticality of Lagrange multipliers in variational systems, which are associated with the class of problems in composite optimization known as extended nonlinear programming (ENLP). The importance of both ENLP and the concept of multiplier criticality in variational systems has been recognized in theoretical and numerical aspects of optimization and variational analysis, while the criticality notion has never been investigated in the ENLP framework. We present here a systematic study of critical and noncritical multipliers in a general vari…
In the paper, we study the stochastic alternating direction method of multipliers (ADMM) for the nonconvex optimizations, and propose three classes of the nonconvex stochastic ADMM with variance reduction, based on different reduced variance stochastic gradients. Specifically, the first class called the nonconvex stochastic variance reduced gradient ADMM (SVRG-ADMM), uses a multi-stage scheme to progressively reduce the variance of stochastic gradients. The second is the nonconvex stochastic average gradient ADMM (SAG-ADMM), which additionally uses the old gradients estimated in the previous i…
In this paper, we consider the problem of reducing network delay in stochastic network utility optimization problems. We start by studying the recently proposed quadratic Lyapunov function based algorithms (QLA). We show that for every stochastic problem, there is a corresponding \emph{deterministic} problem, whose dual optimal solution "exponentially attracts" the network backlog process under QLA. In particular, the probability that the backlog vector under QLA deviates from the attractor is exponentially decreasing in their Euclidean distance. This not only helps to explain how QLA achieves…
In this paper, we study variational inequality problems (VIPs) with involved mappings and feasible sets characterized by polynomial functions (namely, polynomial VIPs). We propose a numerical algorithm for computing solutions to polynomial VIPs based on Lagrange multiplier expressions and the Moment-SOS hierarchy of semidefinite relaxations. We also extend our approach to finding more or even all solutions to polynomial VIPs. We show that the method proposed in this paper can find solutions or detect the nonexistence of solutions within finitely many steps, under some general assumptions. In a…
Solutions to nonlinear, nonconvex optimization problems can fail to satisfy the KKT optimality conditions even when they are optimal. This is due to the fact that unless constraint qualifications (CQ) are satisfied, Lagrange multipliers may fail to exist. Even if the KKT conditions are applicable, the multipliers may not be unique. These possibilities also affect AC optimal power flow (OPF) problems which are routinely solved in power systems planning, scheduling and operations. The complex structure -- in particular the presence of the nonlinear power flow equations which naturally exhibit a …
We examine two central regularization strategies for monotone variational inequalities, the first a direct regularization of the operative monotone mapping, and the second via regularization of the associated dual gap function. A key link in the relationship between the solution sets to these various regularized problems is the idea of exact regularization, which, in turn, is fundamentally associated with the existence of Lagrange multipliers for the regularized variational inequality. A regularization is said to be exact if a solution to the regularized problem is a solution to the unregulari…
We show how a stochastic version of the Lagrange multiplier method can be combined with the stochastic maximum principle for jump diffusions to solve certain constrained stochastic optimal control problems. Two different terminal constraints are considered; one constraint holds in expectation and the other almost surely. As an application of this method, we study the effects of inflation- and wage risk on optimal consumption. To do this, we consider the optimal consumption problem for a budget constrained agent with a Lévy income process and stochastic inflation. The agent must choose a consum…
This paper presents a fully asynchronous and distributed approach for tackling optimization problems in which both the objective function and the constraints may be nonconvex. In the considered network setting each node is active upon triggering of a local timer and has access only to a portion of the objective function and to a subset of the constraints. In the proposed technique, based on the method of multipliers, each node performs, when it wakes up, either a descent step on a local augmented Lagrangian or an ascent step on the local multiplier vector. Nodes realize when to switch from the…
This paper establishes the iteration-complexity of a Jacobi-type non-Euclidean proximal alternating direction method of multipliers (ADMM) for solving multi-block linearly constrained nonconvex programs. The subproblems of this ADMM variant can be solved in parallel and hence the method has great potential to solve large scale multi-block linearly constrained nonconvex programs. Moreover, our analysis allows the Lagrange multiplier to be updated with a relaxation parameter in the interval (0, 2).
Second-order optimality conditions for vector nonlinear programming problems with inequality constraints are studied in this paper. We introduce a new second-order constraint qualification, which includes Mangasarian-Fromovitz constraint qualification as a particular case. We obtain necessary and sufficient conditions for weak efficiency of problems with a second-order pseudoconvex vector objective function and quasiconvex constraints. We also derive Lagrange multiplier characterizations of the solution set of a scalar problem with a second-order pseudoconvex objective function and quasiconvex…
There has been a recent surge of interest in physics-based solvers for combinatorial optimization problems. We present a dynamical solver for the Ising problem that is comprised of a network of coupled parametric oscillators and show that it implements Lagrange multiplier constrained optimization. We show that the pump depletion effect, which is intrinsic to parametric oscillators, enforces binary constraints and enables the system's continuous analog variables to converge to the optimal binary solutions to the optimization problem. Moreover, there is an exact correspondence between the equati…
Consider the minimization of a nonconvex differentiable function over a polyhedron. A popular primal-dual first-order method for this problem is to perform a gradient projection iteration for the augmented Lagrangian function and then update the dual multiplier vector using the constraint residual. However, numerical examples show that this approach can exhibit "oscillation" and may not converge. In this paper, we propose a proximal alternating direction method of multipliers for the multi-block version of this problem. A distinctive feature of this method is the introduction of a "smoothed" (…
We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact which allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of…
This paper deals with the geometric numerical integration of gradient flow and its application to optimization. Gradient flows often appear as model equations of various physical phenomena, and their dissipation laws are essential. Therefore, dissipative numerical methods, which are numerical methods replicating the dissipation law, have been studied in the literature. Recently, Cheng, Liu, and Shen proposed a novel dissipative method, the Lagrange multiplier approach, for gradient flows, which is computationally cheaper than existing dissipative methods. Although their efficacy is numerically…
In this paper, we study a general optimization model, which covers a large class of existing models for many applications in imaging sciences. To solve the resulting possibly nonconvex, nonsmooth and non-Lipschitz optimization problem, we adapt the alternating direction method of multipliers (ADMM) with a general dual step-size to solve a reformulation that contains three blocks of variables, and analyze its convergence. We show that for any dual step-size less than the golden ratio, there exists a computable threshold such that if the penalty parameter is chosen above such a threshold and the…
Constrained optimization problems exist in many domains of science, such as thermodynamics, mechanics, economics, etc. These problems are classically solved with the help of the Lagrange multipliers and the Lagrangian function. However, the disadvantage of this approach is that it artificially increases the dimensionality of the problem. Here, we show that the determinant of the Jacobian of the problem (function to optimize and constraints) is null. This extra equation transforms any equality-constrained optimization problem into a solving problem of same dimension. We also introduced the cons…
On error bounds and noncritical Lagrange multipliers
Johann Radon Institute for Computational and Applied Mathematics（Johann Radon计算与应用数学研究所）；Dresden University of Technology, Institute of Numerical Mathematics（德累斯顿工业大学数值数学研究所）
In this paper, we consider a proximal linearized alternating direction method of multipliers (PL-ADMM) for solving linearly constrained nonconvex and possibly nonsmooth optimization problems. The algorithm is generalized by using variable metric proximal terms in the primal updates and an over-relaxation stepsize in the multiplier update. We prove that the sequence generated by this method is bounded and its limit points are critical points. Under the powerful Kurdyka-{Łojasiewicz} properties we prove that the sequence has a finite length thus converges, and we drive its convergence rates.
Iteratively Linearized Reweighted Alternating Direction Method of Multipliers for a Class of Nonconvex Problems
Department of Mathematics, National University of Defense Technology（国防科技大学数学系）；College of Computer, National University of Defense Technology（国防科技大学计算机学院）；The State Key Laboratory for High Performance Computation, National University of Defense Technology（国防科技大学高性能计算国家重点实验室）；Hunan Key Laboratory for Computation and Simulation in Science and Engineering, School of Mathematics and Computational Science, Xiangtan University（湖南计算与模拟科学工程重点实验室，湘潭大学数学与计算科学学院）
We introduce a generalization of the linearized Alternating Direction Method of Multipliers to optimize a real-valued function $f$ of multiple arguments with potentially multiple constraints $g_\circ$ on each of them. The function $f$ may be nonconvex as long as it is convex in every argument, while the constraints $g_\circ$ need to be convex but not smooth. If $f$ is smooth, the proposed Block-Simultaneous Direction Method of Multipliers (bSDMM) can be interpreted as a proximal analog to inexact coordinate descent methods under constraints. Unlike alternative approaches for joint solvers of m…
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；JD Finance America Corporation（京东金融美国公司）
Alternating direction method of multipliers (ADMM) is a popular optimization tool for the composite and constrained problems in machine learning. However, in many machine learning problems such as black-box attacks and bandit feedback, ADMM could fail because the explicit gradients of these problems are difficult or infeasible to obtain. Zeroth-order (gradient-free) methods can effectively solve these problems due to that the objective function values are only required in the optimization. Recently, though there exist a few zeroth-order ADMM methods, they build on the convexity of objective fu…
