Compressed Sensing (CS) encompasses a broad array of theoretical and applied techniques for recovering signals, given partial knowledge of their coefficients. Its applications span various fields, including mathematics, physics, engineering, and several medical sciences. Motivated by our interest in the mathematics behind Magnetic Resonance Imaging (MRI) and CS, we employ convex analysis techniques to analytically determine equivalents of Lagrange multipliers for optimization problems with inequality constraints, specifically a weighted LASSO with voxel-wise weighting. We investigate this problem under assumptions on the fidelity term $\Vert{Ax-b}\Vert_2^2$, either concerning the sign of its gradient or orthogonality-like conditions of its matrix. To be more precise, we either require the sign of each coordinate of $2(Ax-b)^TA$ to be fixed within a rectangular neighborhood of the origin, with the side lengths of the rectangle dependent on the constraints, or we assume $A^TA$ to be diagonal. The objective of this work is to explore the relationship between Lagrange multipliers and the constraints of a weighted variant of LASSO, specifically in the mentioned cases where this relationship can be computed explicitly. As they scale the regularization terms of the weighted LASSO, Lagrange multipliers serve as tuning parameters for the weighted LASSO, prompting the question of their potential effective use as tuning parameters in applications like MR image reconstruction and denoising. This work represents an initial step in this direction.
This paper deals with the geometric numerical integration of gradient flow and its application to optimization. Gradient flows often appear as model equations of various physical phenomena, and their dissipation laws are essential. Therefore, dissipative numerical methods, which are numerical methods replicating the dissipation law, have been studied in the literature. Recently, Cheng, Liu, and Shen proposed a novel dissipative method, the Lagrange multiplier approach, for gradient flows, which is computationally cheaper than existing dissipative methods. Although their efficacy is numerically…
Based on a characterization of the optimality of a feasible solution of a convex entropy minimization problem, one shows that the feasible solutions obtained using formally the Lagrange multipliers method are optimal.
The concept of uniform tangent sets was introduced and discussed in [3 - Krastanov, Ribarska, SIAM J. Control Optim., 55(3), 2017]. This study is devoted to their further investigation and to generalization of the abstract Lagrange multiplier rule from [3].
In this note, three Lagrange multiplier rules introduced in the literature for set valued optimization problems are compared. A generalization of all three results is given which proves that under rather mild assumptions, $x$ is a weak solution to the constrained problem, if and only if it is a weak solution to the Lagrangian.
We use an extension to the infinite dimension of the rank theorem of the differential calculus to establish a Karush-Huhn-Tucker theorem for optimization problems in Banach spaces. We provide an application to variational problems on bounded processus under equality constraints.
We analyze sequences generated by interior point methods (IPMs) in convex and nonconvex settings. We prove that moving the primal feasibility at the same rate as the barrier parameter $μ$ ensures the Lagrange multiplier sequence remains bounded, provided the limit point of the primal sequence has a Lagrange multiplier. This result does not require constraint qualifications. We also guarantee the IPM finds a solution satisfying strict complementarity if one exists. On the other hand, if the primal feasibility is reduced too slowly, then the algorithm converges to a point of minimal complementar…
This paper proposes tight semidefinite relaxations for polynomial optimization. The optimality conditions are investigated. We show that generally Lagrange multipliers can be expressed as polynomial functions in decision variables over the set of critical points. The polynomial expressions can be determined by linear equations. Based on these expressions, new Lasserre type semidefinite relaxations are constructed for solving polynomial optimization. We show that the hierarchy of new relaxations has finite convergence, or equivalently, the new relaxations are tight for a finite relaxation order…
We discuss first order optimality conditions for geometric optimization problems with Neumann boundary conditions and boundary observation. The methods we develop here are applicable to large classes of state systems or cost functionals. Our approach is based on the implicit parametrization theorem and the use of Hamiltonian systems. It establishes equivalence with a constrained optimal control problem and uses Lagrange multipliers under a new simple constraint qualification. In this setting, general functional variations are performed, that combine topological and boundary variations in a nat…
In this paper we apply an augmented Lagrange method to a class of semilinear elliptic optimal control problems with pointwise state constraints. We show strong convergence of subsequences of the primal variables to a local solution of the original problem as well as weak convergence of the adjoint states and weak* convergence of the multipliers associated to the state constraint. Moreover, we show existence of stationary points in arbitrary small neighborhoods of local solutions of the original problem. Additionally, various numerical results are presented.
The paper concerns the study of criticality of Lagrange multipliers in variational systems that has been recognized in both theoretical and numerical aspects of optimization and variational analysis. In contrast to the previous developments dealing with polyhedral KKT systems and the like, we now focus on general nonpolyhedral systems that are associated, in particular, with problems of conic programming. Developing a novel approach, which is mainly based on advanced techniques and tools of second-order variational analysis and generalized differentiation, allows us to overcome principal chall…
This paper studies bilevel polynomial optimization problems. To solve them, we give a method based on polynomial optimization relaxations. Each relaxation is obtained from the Kurash-Kuhn-Tucker (KKT) conditions for the lower level optimization and the exchange technique for semi-infinite programming. For KKT conditions, Lagrange multipliers are represented as polynomial or rational functions. The Moment-SOS relaxations are used to solve the polynomial optimizattion relaxations. Under some general assumptions, we prove the convergence of the algorithm for solving bilevel polynomial optimizatio…
The paper is devoted to the study and applications of criticality of Lagrange multipliers in variational systems, which are associated with the class of problems in composite optimization known as extended nonlinear programming (ENLP). The importance of both ENLP and the concept of multiplier criticality in variational systems has been recognized in theoretical and numerical aspects of optimization and variational analysis, while the criticality notion has never been investigated in the ENLP framework. We present here a systematic study of critical and noncritical multipliers in a general vari…
Lagrange Multipliers and Duality with Applications to Constrained Support Vector Machine
Fariborz Maseeh Department of Mathematics and Statistics, Portland State University（波特兰州立大学法里博兹·马西数学与统计系）；Department of Statistics and Operations Research, The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校统计与运筹学系）
In this paper, we study variational inequality problems (VIPs) with involved mappings and feasible sets characterized by polynomial functions (namely, polynomial VIPs). We propose a numerical algorithm for computing solutions to polynomial VIPs based on Lagrange multiplier expressions and the Moment-SOS hierarchy of semidefinite relaxations. We also extend our approach to finding more or even all solutions to polynomial VIPs. We show that the method proposed in this paper can find solutions or detect the nonexistence of solutions within finitely many steps, under some general assumptions. In a…
Solutions to nonlinear, nonconvex optimization problems can fail to satisfy the KKT optimality conditions even when they are optimal. This is due to the fact that unless constraint qualifications (CQ) are satisfied, Lagrange multipliers may fail to exist. Even if the KKT conditions are applicable, the multipliers may not be unique. These possibilities also affect AC optimal power flow (OPF) problems which are routinely solved in power systems planning, scheduling and operations. The complex structure -- in particular the presence of the nonlinear power flow equations which naturally exhibit a …
We examine two central regularization strategies for monotone variational inequalities, the first a direct regularization of the operative monotone mapping, and the second via regularization of the associated dual gap function. A key link in the relationship between the solution sets to these various regularized problems is the idea of exact regularization, which, in turn, is fundamentally associated with the existence of Lagrange multipliers for the regularized variational inequality. A regularization is said to be exact if a solution to the regularized problem is a solution to the unregulari…
We show how a stochastic version of the Lagrange multiplier method can be combined with the stochastic maximum principle for jump diffusions to solve certain constrained stochastic optimal control problems. Two different terminal constraints are considered; one constraint holds in expectation and the other almost surely. As an application of this method, we study the effects of inflation- and wage risk on optimal consumption. To do this, we consider the optimal consumption problem for a budget constrained agent with a Lévy income process and stochastic inflation. The agent must choose a consum…
There has been a recent surge of interest in physics-based solvers for combinatorial optimization problems. We present a dynamical solver for the Ising problem that is comprised of a network of coupled parametric oscillators and show that it implements Lagrange multiplier constrained optimization. We show that the pump depletion effect, which is intrinsic to parametric oscillators, enforces binary constraints and enables the system's continuous analog variables to converge to the optimal binary solutions to the optimization problem. Moreover, there is an exact correspondence between the equati…
Second-order optimality conditions for vector nonlinear programming problems with inequality constraints are studied in this paper. We introduce a new second-order constraint qualification, which includes Mangasarian-Fromovitz constraint qualification as a particular case. We obtain necessary and sufficient conditions for weak efficiency of problems with a second-order pseudoconvex vector objective function and quasiconvex constraints. We also derive Lagrange multiplier characterizations of the solution set of a scalar problem with a second-order pseudoconvex objective function and quasiconvex…
Constrained optimization problems exist in many domains of science, such as thermodynamics, mechanics, economics, etc. These problems are classically solved with the help of the Lagrange multipliers and the Lagrangian function. However, the disadvantage of this approach is that it artificially increases the dimensionality of the problem. Here, we show that the determinant of the Jacobian of the problem (function to optimize and constraints) is null. This extra equation transforms any equality-constrained optimization problem into a solving problem of same dimension. We also introduced the cons…
On error bounds and noncritical Lagrange multipliers
Johann Radon Institute for Computational and Applied Mathematics（Johann Radon计算与应用数学研究所）；Dresden University of Technology, Institute of Numerical Mathematics（德累斯顿工业大学数值数学研究所）
In this paper we consider three minimization problems, namely quadratic, $ρ$-convex and quadratic fractional programing problems. The quadratic problem is considered with quadratic inequality constraints with bounded continuous and discrete mixed variables. The $ρ$-convex problem is considered with $ρ$-convex inequality constraints in mixed variables. The quadratic fractional problem is studied with quadratic fractional constraints in mixed variables. For all three problems we reformulate the problem as a mathematical programming problem and apply standard Karush Kuhn Tucker necessary conditio…
We provide a modified augmented Lagrange method coupled with a Tikhonov regularization for solving ill-posed state-constrained elliptic optimal control problems with sparse controls. We consider a linear quadratic optimal control problem without any additional $L^2$ regularization terms. The sparsity is guaranteed by an additional $L^1$ term. Here, the modification of the classical augmented Lagrange method guarantees us uniform boundedness of the multiplier that corresponds to the state constraints. We present a coupling between the regularization parameter introduced by the Tikhonov regulari…
