This paper investigates the convex optimization problem with general convex inequality constraints. To cope with this problem, a discrete-time algorithm, called augmented primal-dual gradient algorithm (Aug-PDG), is studied and analyzed. It is shown that Aug-PDG can converge semi-globally to the optimizer at a linear rate under some mild assumptions, such as the quadratic gradient growth condition for the objective function, which is strictly weaker than strong convexity. To our best knowledge, this paper is the first to establish a linear convergence for the studied problem in the discrete-time setting, where an explicit bound is provided for the stepsize. Finally, a numerical example is presented to illustrate the efficacy of the theoretical finding.
Semi-Global Exponential Stability of Augmented Primal-Dual Gradient Dynamics for Constrained Convex Optimization
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Department of Computing and Mathematical Sciences, California Institute of Technology（加州理工学院计算与数学科学系）
We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hetenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality cons…
A conditional-gradient-based single-loop augmented Lagrangian method for inequality constrained optimization
School of Mathematical Sciences, South China Normal University（华南师范大学数学科学学院）；Department of Applied Mathematics, The Hong Kong Polytechnic University（香港理工大学应用数学系）；Department of Mathematics and Statistics, James Madison University（詹姆斯麦迪逊大学数学与统计系）
The augmented Lagrangian method (ALM) is a benchmark for convex programming problems with linear constraints; ALM and its variants for linearly equality-constrained convex minimization models have been well studied in the literature. However, much less attention has been paid to ALM for efficiently solving linearly inequality-constrained convex minimization models. In this paper, we exploit an enlightening reformulation of the newly developed indefinite linearized ALM for the equality-constrained convex optimization problem, and present a new indefinite linearized ALM scheme for efficiently so…
In this paper, we propose a unified primal-dual algorithm framework based on the augmented Lagrangian function for composite convex problems with conic inequality constraints. The new framework is highly versatile. First, it not only covers many existing algorithms such as PDHG, Chambolle-Pock (CP), GDA, OGDA and linearized ALM, but also guides us to design a new efficient algorithm called Simi-OGDA (SOGDA). Second, it enables us to study the role of the augmented penalty term in the convergence analysis. Interestingly, a properly selected penalty not only improves the numerical performance of…
We propose a variant of the classical augmented Lagrangian method for constrained optimization problems in Banach spaces. Our theoretical framework does not require any convexity or second-order assumptions and allows the treatment of inequality constraints with infinite-dimensional image space. Moreover, we discuss the convergence properties of our algorithm with regard to feasibility, global optimality, and KKT conditions. Some numerical results are given to illustrate the practical viability of the method.
Shape-constrained convex regression problem deals with fitting a convex function to the observed data, where additional constraints are imposed, such as component-wise monotonicity and uniform Lipschitz continuity. This paper provides a unified framework for computing the least squares estimator of a multivariate shape-constrained convex regression function in $\mathbb{R}^d$. We prove that the least squares estimator is computable via solving an essentially constrained convex quadratic programming (QP) problem with $(d+1)n$ variables, $n(n-1)$ linear inequality constraints and $n$ possibly non…
An Augmented Lagrangian Approach to Conically Constrained Non-monotone Variational Inequality Problems
National Center for Translational Medicine, Shanghai Jiao Tong University（上海交通大学国家转化医学中心）；Sino-US Global Logistics Institute, Shanghai Jiao Tong University（上海交通大学中美物流研究院）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；University of Minnesota（明尼苏达大学）
We consider the convex minimization model with both linear equality and inequality constraints, and reshape the classic augmented Lagrangian method (ALM) by balancing its subproblems. As a result, one of its subproblems decouples the objective function and the coefficient matrix without any extra condition, and the other subproblem becomes a positive definite system of linear equations or a positive definite linear complementary problem. The balanced ALM advances the classic ALM by enlarging its applicable range, balancing its subproblems, and improving its implementation. We also extend our d…
We develop a decomposition method based on the augmented Lagrangian framework to solve a broad family of semidefinite programming problems, possibly with nonlinear objective functions, nonsmooth regularization, and general linear equality/inequality constraints. In particular, the positive semidefinite variable along with a group of linear constraints can be transformed into a variable on a smooth manifold via matrix factorization. The nonsmooth regularization and other general linear constraints are handled by the augmented Lagrangian method. Therefore, each subproblem can be solved by a semi…
Optimality conditions in the form of a variational inequality are proved for a class of constrained optimal control problems of stochastic differential equations. The cost function and the inequality constraints are functions of the probability distribution of the state variable at the final time. The analysis uses in an essential manner a convexity property of the set of reachable probability distributions. An augmented Lagrangian method based on the obtained optimality conditions is proposed and analyzed for solving iteratively the problem. At each iteration of the method, a standard stochas…
We propose a distributed solution for a constrained convex optimization problem over a network of clustered agents each consisted of a set of subagents. The communication range of the clustered agents is such that they can form a connected undirected graph topology. The total cost in this optimization problem is the sum of the local convex costs of the subagents of each cluster. We seek a minimizer of this cost subject to a set of affine equality constraints, and a set of affine inequality constraints specifying the bounds on the decision variables if such bounds exist. We design our distribut…
In this paper, we aim to solve high dimensional convex quadratic programming (QP) problems with a large number of quadratic terms, linear equality and inequality constraints. In order to solve the targeted {\bf QP} problems to a desired accuracy efficiently, we develop a two-phase {\bf P}roximal {\bf A}ugmented {\bf L}agrangian method {(QPPAL)}, with Phase I to generate a reasonably good initial point to warm start Phase II to obtain an accurate solution efficiently. More specifically, in Phase I, based on the recently developed symmetric Gauss-Seidel (sGS) decomposition technique, we design a…
Argonne National Laboratory（阿贡国家实验室）；The University of Chicago（芝加哥大学）；University of California, Berkeley（加州大学伯克利分校）；International Computer Science Institute（国际计算机科学研究所）
We study nonlinear optimization problems with a stochastic objective and deterministic equality and inequality constraints, which emerge in numerous applications including finance, manufacturing, power systems and, recently, deep neural networks. We propose an active-set stochastic sequential quadratic programming (StoSQP) algorithm that utilizes a differentiable exact augmented Lagrangian as the merit function. The algorithm adaptively selects the penalty parameters of the augmented Lagrangian and performs a stochastic line search to decide the stepsize. The global convergence is established:…
This paper analyzes the contraction of the primal-dual gradient optimization via contraction theory in the context of discrete-time updating dynamics. The contraction theory based on Riemannian manifolds is first established for convergence analysis of a convex optimization algorithm. The equality and inequality constrained optimization cases are studied, respectively. Under some reasonable assumptions, we construct the Riemannian metric to characterize a contraction region. It is shown that if the step-sizes of the updating dynamics are properly designed, the convergence rates for both cases …
Projecting the distance measures onto a low-dimensional space is an efficient way of mitigating the curse of dimensionality in the classical Wasserstein distance using optimal transport. The obtained maximized distance is referred to as projection robust Wasserstein (PRW) distance. In this paper, we equivalently reformulate the computation of the PRW distance as an optimization problem over the Cartesian product of the Stiefel manifold and the Euclidean space with additional nonlinear inequality constraints. We propose a Riemannian exponential augmented Lagrangian method (ReALM) with a global …
For any given neural network architecture a permutation of weights and biases results in the same functional network. This implies that optimization algorithms used to `train' or `learn' the network are faced with a very large number (in the millions even for small networks) of equivalent optimal solutions in the parameter space. To the best of our knowledge, this observation is absent in the literature. In order to narrow down the parameter search space, a novel technique is introduced in order to fix the bias vector configurations to be monotonically increasing. This is achieved by augmentin…
This paper considers the problem of minimizing a convex expectation function with a set of inequality convex expectation constraints. We present a computable stochastic approximation type algorithm, namely the stochastic linearized proximal method of multipliers, to solve this convex stochastic optimization problem. This algorithm can be roughly viewed as a hybrid of stochastic approximation and the traditional proximal method of multipliers. Under mild conditions, we show that this algorithm exhibits $O(K^{-1/2})$ expected convergence rates for both objective reduction and constraint violatio…
We study distributed algorithms for seeking a Nash equilibrium in a class of non-cooperative convex games with strongly monotone mappings. Each player has access to her own smooth local cost function and can communicate to her neighbors in some undirected graph. To deal with fast distributed learning of Nash equilibria under such settings, we introduce a so called augmented game mapping and provide conditions under which this mapping is strongly monotone. We consider a distributed gradient play algorithm for determining a Nash equilibrium (GRANE). The algorithm involves every player performing…
For a class of nonsmooth composite optimization problems with linear equality constraints, we utilize a Lyapunov-based approach to establish the global exponential stability of the primal-dual gradient flow dynamics based on the proximal augmented Lagrangian. The result holds when the differentiable part of the objective function is strongly convex with a Lipschitz continuous gradient; the non-differentiable part is proper, lower semi-continuous, and convex; and the matrix in the linear constraint is full row rank. Our quadratic Lyapunov function generalizes recent result from strongly convex …
This thesis presents new mathematical algorithms for the numerical solution of a mathematical problem class called \emph{dynamic optimization problems}. These are mathematical optimization problems, i.e., problems in which numbers are sought that minimize an expression subject to obeying equality and inequality constraints. Dynamic optimization problems are distinct from non-dynamic problems in that the sought numbers may vary over one independent variable. This independent variable can be thought of as, e.g., time. This thesis presents three methods, with emphasis on algorithms, convergence a…
Optimization theory in Banach spaces suffers from the lack of available constraint qualifications. Despite the fact that there exist only a very few constraint qualifications, they are, in addition, often violated even in simple applications. This is very much in contrast to finite-dimensional nonlinear programs, where a large number of constraint qualifications is known. Since these constraint qualifications are usually defined using the set of active inequality constraints, it is difficult to extend them to the infinite-dimensional setting. One exception is a recently introduced sequential c…
BR-MPPI: Barrier-Rate Guided MPPI for Enforcing Multiple Inequality Constraints with Learned Signed Distance Fields
Robotics Department, University of Michigan（密歇根大学机器人系）；Contextual Robotics Institute, University of California San Diego（加州大学圣地亚哥分校情境机器人研究所）；Toyota Motor North America, Research & Development（丰田北美公司研发部）
Nonconvex Composite Functional Constraints via First-Order Augmented Lagrangian Methods under Local Regularity
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, USA（赫尔曼·米勒工程与系统工程学院，佐治亚理工学院，美国亚特兰大，GA）；Sauder School of Business, University of British Columbia, Vancouver, BC, Canada（萨德勒商学院，不列颠哥伦比亚大学，加拿大温哥华，BC）
We consider optimization problems on Riemannian manifolds with equality and inequality constraints, which we call Riemannian nonlinear optimization (RNLO) problems. Although they have numerous applications, the existing studies on them are limited especially in terms of algorithms. In this paper, we propose Riemannian sequential quadratic optimization (RSQO) that uses a line-search technique with an ell_1 penalty function as an extension of the standard SQO algorithm for constrained nonlinear optimization problems in Euclidean spaces to Riemannian manifolds. We prove its global convergence to …
We consider optimization problems on manifolds with equality and inequality constraints. A large body of work treats constrained optimization in Euclidean spaces. In this work, we consider extensions of existing algorithms from the Euclidean case to the Riemannian case. Thus, the variable lives on a known smooth manifold and is further constrained. In doing so, we exploit the growing literature on unconstrained Riemannian optimization. For the special case where the manifold is itself described by equality constraints, one could in principle treat the whole problem as a constrained problem in …
Semidefinite programs are generally challenging to solve due to their high dimensionality. Burer and Monteiro developed a non-convex approach to solve linear SDP problems by applying its low rank property. Their approach is fast because they used factorization to reduce the problem size. In this paper, we focus on solving the SDP relaxation of a graph equipartition problem, which involves an additional semidefinite upper bound constraint over the traditional linear SDP. By applying the factorization approach, we get a non-convex problem with an additional non-smooth spectral inequality constra…
This paper develops and analyzes feedback-based online optimization methods to regulate the output of a linear time-invariant (LTI) dynamical system to the optimal solution of a time-varying convex optimization problem. The design of the algorithm is based on continuous-time primal-dual dynamics, properly modified to incorporate feedback from the LTI dynamical system, applied to a proximal augmented Lagrangian function. The resultant closed-loop algorithm tracks the solution of the time-varying optimization problem without requiring knowledge of (time-varying) disturbances in the dynamical sys…
