In this article we consider a convex feasible set described by inequality constraints that are continuous and not necessarily Lipschitz or convex. We show that if the Slater constraint qualification and a non-degeneracy condition are satisfied, then the Karush-Kuhn-Tucker type optimality condition is both necessary and sufficient. In this way we extend previous results which are proved for Lipschitz and differentiable inequalities.
Primal-Dual Error Bounds and KKT Metric Subregularity in Convex Composite Optimization
School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学院）；Beijing Key Laboratory on MCAACI/Key Laboratory of Mathematical Theory and Computation in Information Security, Beijing Institute of Technology（北京理工大学计算机辅助设计与图形学北京市重点实验室/信息安全数学理论与计算北京市重点实验室）
It was proved in [14] that the existence of a noncritical multiplier for a (smooth) nonlinear programming problem is equivalent to an error bound condition for the Karush-Kuhn-Thcker (KKT) system without any assumptions. This paper investigates whether this result still holds true for a (smooth) nonlinear semidefinite programming (SDP) problem. We first introduce the notion of critical and noncritical multipliers for a SDP problem and obtain their complete characterizations in terms of the problem data. We prove for the SDP problem, the noncriticality property can be derived from the error bou…
Evaluation complexity for convexly constrained optimization is considered and it is shown first that the complexity bound of $O(ε^{-3/2})$ proved by Cartis, Gould and Toint (IMAJNA 32(4) 2012, pp.1662-1695) for computing an $ε$-approximate first-order critical point can be obtained under significantly weaker assumptions. Moreover, the result is generalized to the case where high-order derivatives are used, resulting in a bound of $O(ε^{-(p+1)/p})$ evaluations whenever derivatives of order $p$ are available. It is also shown that the bound of $O(ε_P^{-1/2}ε_D^{-3/2})$ evaluations ($ε_P$ and $ε_…
This work concerns the local convergence theory of Newton and quasi-Newton methods for convex-composite optimization: minimize f(x):=h(c(x)), where h is an infinite-valued proper convex function and c is C^2-smooth. We focus on the case where h is infinite-valued piecewise linear-quadratic and convex. Such problems include nonlinear programming, mini-max optimization, estimation of nonlinear dynamics with non-Gaussian noise as well as many modern approaches to large-scale data analysis and machine learning. Our approach embeds the optimality conditions for convex-composite optimization problem…
Second-Order KKT Guarantees for Bregman ADMM in Nonconvex and Non-Lipschitz Optimization
Department of Electrical and Computer Engineering, Iowa State University（电气与计算机工程系，爱荷华州立大学）；Department of Computer Science and Engineering, Ohio State University（计算机科学与工程系，俄亥俄州立大学）；DAMO Academy, Alibaba Group US（阿里巴巴集团美国DAMO研究院）
In the recent paper of Giorgi, Jiménez and Novo (J Optim Theory Appl 171:70--89, 2016), the authors introduced the so-called approximate Karush-Kuhn-Tucker (AKKT) condition for smooth multiobjective optimization problems and obtained some AKKT-type necessary optimality conditions and sufficient optimality conditions for weak efficient solutions of such a problem. In this note, we extend these optimality conditions to locally Lipschitz multiobjective optimization problems using Mordukhovich subdifferentials. Furthermore, we prove that, under some suitable additional conditions, an AKKT conditio…
University of Waterloo（滑铁卢大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Statistics and Actuarial Science（统计与精算科学系）；David R. Cheriton School of Computer Science（大卫·R·谢里顿计算机科学学院）；Centre for Pattern Analysis and Machine Intelligence（模式分析与机器智能中心）；Machine Learning Laboratory（机器学习实验室）；Data Analytics Laboratory（数据分析实验室）
This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start with some preliminaries on properties of sets, norms, functions, and concepts of optimization. Then, we introduce the optimization problem, standard optimization problems (including linear programming, quadratic programming, and semidefinite programming), and convex problems. We also introduce some techniques such as eliminating inequality, equality, and set constraints, adding sl…
We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hetenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality cons…
Nonconvex Composite Functional Constraints via First-Order Augmented Lagrangian Methods under Local Regularity
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, USA（赫尔曼·米勒工程与系统工程学院，佐治亚理工学院，美国亚特兰大，GA）；Sauder School of Business, University of British Columbia, Vancouver, BC, Canada（萨德勒商学院，不列颠哥伦比亚大学，加拿大温哥华，BC）
Southern Methodist University（南卫理公会大学）；Shanghai university of Finance & Economics（上海财经大学）；Georgia Tech（佐治亚理工学院）
Nonconvex sparse models have received significant attention in high-dimensional machine learning. In this paper, we study a new model consisting of a general convex or nonconvex objectives and a variety of continuous nonconvex sparsity-inducing constraints. For this constrained model, we propose a novel proximal point algorithm that solves a sequence of convex subproblems with gradually relaxed constraint levels. Each subproblem, having a proximal point objective and a convex surrogate constraint, can be efficiently solved based on a fast routine for projection onto the surrogate constraint. W…
Study about theory and algorithms for constrained optimization usually assumes that the feasible region of the optimization problem is nonempty. However, there are many important practical optimization problems whose feasible regions are not known to be nonempty or not, and optimizers of the objective function with the least constraint violation prefer to be found. A natural way for dealing with these problems is to extend the constrained optimization problem as the one optimizing the objective function over the set of points with the least constraint violation. Firstly, the minimization probl…
We present a unified study of first and second order necessary and sufficient optimality conditions for minimax and Chebyshev optimisation problems with cone constraints. First order optimality conditions for such problems can be formulated in several different forms: in terms of a linearised problem, in terms of Lagrange multipliers (KKT-points), in terms of subdifferentials and normal cones, in terms of a nonsmooth penalty function, in terms of cadres with positive cadre multipliers, and in an alternance form. We describe interconnections between all these forms of necessary and sufficient o…
We propose a new algorithm to solve optimization problems of the form $\min f(X)$ for a smooth function $f$ under the constraints that $X$ is positive semidefinite and the diagonal blocks of $X$ are small identity matrices. Such problems often arise as the result of relaxing a rank constraint (lifting). In particular, many estimation tasks involving phases, rotations, orthonormal bases or permutations fit in this framework, and so do certain relaxations of combinatorial problems such as Max-Cut. The proposed algorithm exploits the facts that (1) such formulations admit low-rank solutions, and …
In this paper we consider a class of structured nonsmooth difference-of-convex (DC) constrained DC program in which the first convex component of the objective and constraints is the sum of a smooth and nonsmooth functions while their second convex component is the supremum of finitely many convex smooth functions. The existing methods for this problem usually have a weak convergence guarantee or require a feasible initial point. Inspired by the recent work (Math Oper. Res. 42(1):95--118, 2017 by Pang et al.), in this paper we propose two infeasible methods with strong convergence guarantee fo…
Projection-Free Functional Constrained Optimization for Risk Aversion and Sparsity Control
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology（赫尔曼·米利顿·斯图尔特工业与系统工程学院，佐治亚理工学院）；College of Management of Technology, EPFL（技术管理学院，苏黎世联邦理工学院）
Alternating Direction Method of Multipliers for nonlinear constrained convex problems and applications to distributed resource allocation and constrained machine learning
Motivated by robotic trajectory optimization problems we consider the Augmented Lagrangian approach to constrained optimization. We first propose an alternative augmentation of the Lagrangian to handle the inequality case (not based on slack variables) and a corresponding "central" update of the dual parameters. We proove certain properties of this update: roughly, in the case of LPs and when the "constraint activity" does not change between iterations, the KKT conditions hold after just one iteration. This gives essential insight on when the method is efficient in practise. We then present ou…
This paper proposes low-complexity algorithms for finding approximate second-order stationary points (SOSPs) of problems with smooth non-convex objective and linear constraints. While finding (approximate) SOSPs is computationally intractable, we first show that generic instances of the problem can be solved efficiently. More specifically, for a generic problem instance, certain strict complementarity (SC) condition holds for all Karush-Kuhn-Tucker (KKT) solutions (with probability one). The SC condition is then used to establish an equivalence relationship between two different notions of SOS…
Georgia Institute of Technology（佐治亚理工学院）；Shanghai University of Finance and Economics（上海财经大学）
Functional constrained optimization is becoming more and more important in machine learning and operations research. Such problems have potential applications in risk-averse machine learning, semisupervised learning, and robust optimization among others. In this paper, we first present a novel Constraint Extrapolation (ConEx) method for solving convex functional constrained problems, which utilizes linear approximations of the constraint functions to define the extrapolation (or acceleration) step. We show that this method is a unified algorithm that achieves the best-known rate of convergence…
