Given a non-convex optimization problem, we study conditions under which every Karush-Kuhn-Tucker (KKT) point is a global optimizer. This property is known as KT-invexity and allows to identify the subset of problems where an interior point method always converges to a global optimizer. In this work, we provide necessary conditions for KT-invexity in n-dimensions and show that these conditions become sufficient in the two-dimensional case. As an application of our results, we study the Optimal Power Flow problem, showing that under mild assumptions on the variable's bounds, our new necessary and sufficient conditions are met for problems with two degrees of freedom.
In this paper we obtain second- and first-order optimality conditions of Kuhn-Tucker type and Fritz John one for weak efficiency in the vector problem with inequality constraints. In the necessary conditions we suppose that the objective function and the active constraints are continuously differentiable. We introduce notions of KTSP-invex problem and second-order KTSP-invex one. We obtain that the vector problem is (second-order) KTSP-invex if and only if for every triple $(\bar x,\barλ,\barμ)$ with Lagrange multipliers $\barλ$ and $\barμ$ for the objective function and constraints respective…
LEARN: An Invex Loss for Outlier Oblivious Robust Online Optimization
Institute of Data Science（数据科学研究所）；National University of Singapore（新加坡国立大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Mathematics（数学系）
We define w-invex set, w-preinvex, w-strictly preinvex, w-quasi preinvex, w-strictly quasi preinvex, w-semi-strictly quasi preinvex, and w-pre pseudo-invex functions in this context. And these form a class of real functions, which is the generalization of a family of preinvex functions. Here, we provide a thorough analysis of the core characteristics of these functions, along with numerous examples that help to illustrate the idea. Finally, w-quasi preinvex, w-strictly preinvex, and w-strictly quasi preinvex functions are used to analyze the optimization problems.
We are interested in local quasi efficient solutions for nonsmooth vector optimization problems under new generalized approximate invexity assumptions. We formulate necessary and sufficient optimality conditions based on Stampacchia and Minty types of vector variational inequalities involving Clarke's generalized Jacobians. We also establish the relationship between local quasi weak efficient solutions and vector critical points.
University of Queensland（昆士兰大学）；The Centre for Information Resilience（信息弹性中心）；International Computer Science Institute（国际计算机科学研究所）；Stanford University（斯坦福大学）；University of California at Berkeley（加州大学伯克利分校）
We consider a variant of inexact Newton Method, called Newton-MR, in which the least-squares sub-problems are solved approximately using Minimum Residual method. By construction, Newton-MR can be readily applied for unconstrained optimization of a class of non-convex problems known as invex, which subsumes convexity as a sub-class. For invex optimization, instead of the classical Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global convergence can be guaranteed under a weaker notion of joint regularity of Hessian and gradient. We also obtain Newton-MR's problem-independ…
With this note we bring again into attention a vector dual problem neglected by the contributions who have recently announced the successful healing of the trouble encountered by the classical duals to the classical linear vector optimization problem. This vector dual problem has, different to the mentioned works which are of set-valued nature, a vector objective function. Weak, strong and converse duality for this "new-old" vector dual problem are proven and we also investigate its connections to other vector duals considered in the same framework in the literature. We also show that the effi…
The synthesis problem of static output feedback controllers within the anistropic-norm setup is revisited. A tractable synthesis approach involving iterations over a convex optimization problem is suggested, similarly to existing results for the Hinfinity-norm minimization case. The results are formulated by a couple of Linear Matrix Inequalities coupled via a bilinear equality, revealing, as in the Hinfinity case, the duality between the control-type and filtering type LMIs and allowing a tractable iterative method to cope with practical static output feedback synthesis problems. The resultin…
In this short survey, I revisit the role of the proximal point method in large scale optimization. I focus on three recent examples: a proximally guided subgradient method for weakly convex stochastic approximation, the prox-linear algorithm for minimizing compositions of convex functions and smooth maps, and Catalyst generic acceleration for regularized Empirical Risk Minimization.
We revisit the operator splitting schemes proposed in a recent work of [Some extensions of the operator splitting schemes based on Lagrangian and primal-dual: A unified proximal point analysis, Feng Xue, Optimization, 2022, doi: 10.1080/02331934.2022.2057309], and further analyze the convergence of the generalized Bregman distance and the primal-dual gap of these algorithms within a unified proximal point framework. The possibility of reduction to a simple resolvent is also discussed by exploiting the structure and possible degeneracy of the underlying metric.
Redwood Center for Theoretical Neuroscience（雷德伍德理论神经科学中心）；University of California, Berkeley（加州大学伯克利分校）
Understanding of the behavior of algorithms for resolving the optimization problem (hereafter shortened to OP) of optimizing a differentiable loss function (OP1), is enhanced by knowledge of the critical points of that loss function, i.e. the points where the gradient is 0. Here, we describe a solution to the problem of finding critical points by proposing and solving three optimization problems: 1) minimizing the norm of the gradient (OP2), 2) minimizing the difference between the pre-conditioned update direction and the gradient (OP3), and 3) minimizing the norm of the gradient along the upd…
Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Grenoble INP（格勒诺布尔国立理工学院）；Criteo AI Lab（Criteo人工智能实验室）
We propose a new family of adaptive first-order methods for a class of convex minimization problems that may fail to be Lipschitz continuous or smooth in the standard sense. Specifically, motivated by a recent flurry of activity on non-Lipschitz (NoLips) optimization, we consider problems that are continuous or smooth relative to a reference Bregman function - as opposed to a global, ambient norm (Euclidean or otherwise). These conditions encompass a wide range of problems with singular objectives, such as Fisher markets, Poisson tomography, D-design, and the like. In this setting, the applica…
Zuse Institute Berlin（柏林祖斯研究所）；Technische Universität Berlin（柏林工业大学）；Alibaba Inc（阿里巴巴公司）
We revisit the concept of "adversary" in online learning, motivated by solving robust optimization and adversarial training using online learning methods. While one of the classical setups in online learning deals with the "adversarial" setup, it appears that this concept is used less rigorously, causing confusion in applying results and insights from online learning. Specifically, there are two fundamentally different types of adversaries, depending on whether the "adversary" is able to anticipate the exogenous randomness of the online learning algorithms. This is particularly relevant to rob…
We study a general convex optimization problem, which covers various classic problems in different areas and particularly includes many optimal transport related problems arising in recent years. To solve this problem, we revisit the classic Bregman proximal point algorithm (BPPA) and introduce a new inexact stopping condition for solving the subproblems, which can circumvent the underlying feasibility difficulty often appearing in existing inexact conditions when the problem has a complex feasible set. Our inexact condition also covers several existing inexact conditions as special cases and …
In this short expository note, we describe a unified algorithmic perspective on several classical problems which have traditionally been studied in different communities. This perspective views the main characters -- the problems of Optimal Transport, Minimum Mean Cycle, Matrix Scaling, and Matrix Balancing -- through the same lens of optimization problems over joint probability distributions P(x,y) with constrained marginals. While this is how Optimal Transport is typically introduced, this lens is markedly less conventional for the other three problems. This perspective leads to a simple and…
We study distributed multiagent optimization over (directed, time-varying) graphs. We consider the minimization of $F+G$ subject to convex constraints, where $F$ is the smooth strongly convex sum of the agent's losses and $G$ is a nonsmooth convex function. We build on the SONATA algorithm: the algorithm employs the use of surrogate objective functions in the agents' subproblems (going thus beyond linearization, such as proximal-gradient) coupled with a perturbed (push-sum) consensus mechanism that aims to track locally the gradient of $F$. SONATA achieves precision $ε>0$ on the objective valu…
The problem of finding the zeros of the sum of two maximally monotone operators is of fundamental importance in optimization and variational analysis. In this paper, we systematically study Attouch-Théra duality for this problem. We provide new results related to Passty's parallel sum, to Eckstein and Svaiter's extended solution set, and to Combettes' fixed point description of the set of primal solutions. Furthermore, paramonotonicity is revealed to be a key property because it allows for the recovery of all primal solutions given just one arbitrary dual solution. As an application, we genera…
Google Research（谷歌研究院）；University of Washington（华盛顿大学）
We consider the problem of minimizing a non-convex objective while preserving the privacy of the examples in the training data. Building upon the previous variance-reduced algorithm SpiderBoost, we introduce a new framework that utilizes two different kinds of gradient oracles. The first kind of oracles can estimate the gradient of one point, and the second kind of oracles, less precise and more cost-effective, can estimate the gradient difference between two points. SpiderBoost uses the first kind periodically, once every few steps, while our framework proposes using the first oracle whenever…
Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Chinese University of Hong Kong（香港中文大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Distributed optimization has received a lot of interest in recent years due to its wide applications in various fields. In this work, we revisit the convergence property of the decentralized gradient descent [A. Nedi{ć}-A.Ozdaglar (2009)] on the whole space given by $$ x_i(t+1) = \sum^m_{j=1}w_{ij}x_j(t) - α(t) \nabla f_i(x_i(t)), $$ where the stepsize is given as $α(t) = \frac{a}{(t+w)^p}$ with $0< p\leq 1$. Under the strongly convexity assumption on the total cost function $f$ with local cost functions $f_i$ not necessarily being convex, we show that the sequence converges to the optimizer w…
The orienteering problem is a route optimization problem which consists in finding a simple cycle that maximizes the total collected profit subject to a maximum distance limitation. In the last few decades, the occurrence of this problem in real-life applications has boosted the development of many heuristic algorithms to solve it. However, during the same period, not much research has been devoted to the field of exact algorithms for the orienteering problem. The aim of this work is to develop an exact method which is able to obtain optimality certification in a wider set of instances than wi…
University of Insubria（因苏布里亚大学）；Federico Santa María University（费德里科·圣玛丽亚大学）；University of Florence（佛罗伦萨大学）；University of Bologna（博洛尼亚大学）
Training a Support Vector Machine (SVM) requires the solution of a quadratic programming problem (QP) whose computational complexity becomes prohibitively expensive for large scale datasets. Traditional optimization methods cannot be directly applied in these cases, mainly due to memory restrictions. By adopting a slightly different objective function and under mild conditions on the kernel used within the model, efficient algorithms to train SVMs have been devised under the name of Core Vector Machines (CVMs). This framework exploits the equivalence of the resulting learning problem with the …
