We present a general technique for the analysis of first-order methods. The technique relies on the construction of a duality gap for an appropriate approximation of the objective function, where the function approximation improves as the algorithm converges. We show that in continuous time enforcement of an invariant that this approximate duality gap decreases at a certain rate exactly recovers a wide range of first-order continuous-time methods. We characterize the discretization errors incurred by different discretization methods, and show how iteration-complexity-optimal methods for various classes of problems cancel out the discretization error. The techniques are illustrated on various classes of problems -- including convex minimization for Lipschitz-continuous objectives, smooth convex minimization, composite minimization, smooth and strongly convex minimization, solving variational inequalities with monotone operators, and convex-concave saddle-point optimization -- and naturally extend to other settings.
The aim of this paper is to revisit some duality results in conic linear programming and to answer an open problem related to the duality gap function for Gale's example.
In this paper we show that a convexifiability property of nonconvex quadratic programs with nonnegative variables and quadratic constraints guarantees zero duality gap between the quadratic programs and their semi-Lagrangian duals. More importantly, we establish that this convexifiability is hidden in classes of nonnegative homogeneous quadratic programs and discrete quadratic programs, such as mixed integer quadratic programs, revealing zero duality gaps. As an application, we prove that robust counterparts of uncertain mixed integer quadratic programs with objective data uncertainty enjoy ze…
Using the Attouch-Théra duality, we study the cycles, gap vectors and fixed point sets of compositions of proximal mappings. Sufficient conditions are given for the existence of cycles and gap vectors. A primal-dual framework provides an exact relationship between the cycles and gap vectors. We also introduce the generalized cycle and gap vectors to tackle the case when the classical ones do not exist. Examples are given to illustrate our results.
This paper provides a duality gap convergence analysis for the standard ADMM as well as a linearized version of ADMM. It is shown that under appropriate conditions, both methods achieve linear convergence. However, the standard ADMM achieves a faster accelerated convergence rate than that of the linearized ADMM. A simple numerical example is used to illustrate the difference in convergence behavior.
Based on concepts like kth convex hull and finer characterization of nonconvexity of a function, we propose a refinement of the Shapley-Folkman lemma and derive a new estimate for the duality gap of nonconvex optimization problems with separable objective functions. We apply our result to a network flow problem and the dynamic spectrum management problem in communication systems as examples to demonstrate that the new bound can be qualitatively tighter than the existing ones. The idea is also applicable to cases with general nonconvex constraints.
Classical primal-dual affine programming takes place over finite dimensional real vector spaces. This results in beautiful duality theory, connecting the optimal solu- tions of the primal maximization problem and the dual minimization problems. These results include the Existence Duality Theorem, which guarantees optimal solutions to any feasible bounded program; and the Strong Duality Theorem, which implies that optimal solutions for primal and dual programs must have the same objective value. In a common extension of classical affine programming, we see that the Strong Duality does not hold …
We survey research that studies the connection between the computational complexity of optimization problems on the one hand, and the duality gap between the primal and dual optimization problems on the other. To our knowledge, this is the first survey that connects the two very important areas. We further look at a similar phenomenon in finite model theory relating to complexity and optimization.
Training deep neural networks is a challenging non-convex optimization problem. Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently provided an equivalent convex training problem. However, extending this result to deeper networks remains to be an open problem. In this paper, we prove that the duality gap for deeper linear networks with vector outputs is non-zero. In contrast, we show that the zero duality gap can be obtained by stacking standard deep networks in parallel, which we call a paral…
We consider the problem of finding a low rank symmetric matrix satisfying a system of linear equations, as appears in phase retrieval. In particular, we solve the gauge dual formulation, but use a fast approximation of the spectral computations to achieve a noisy solution estimate. This estimate is then used as the initialization of an alternating gradient descent scheme over a nonconvex rank-1 matrix factorization formulation. Numerical results on small problems show consistent recovery, with very low computational cost.
UW-Madison（威斯康星大学麦迪逊分校）；University of Chicago（芝加哥大学）
This note provides a novel, simple analysis of the method of conjugate gradients for the minimization of convex quadratic functions. In contrast with standard arguments, our proof is entirely self-contained and does not rely on the existence of Chebyshev polynomials. Another advantage of our development is that it clarifies the relation between the method of conjugate gradients and general accelerated methods for smooth minimization by unifying their analyses within the framework of the Approximate Duality Gap Technique that was introduced by the authors.
We study conjugate and Lagrange dualities for composite optimization problems within the framework of abstract convexity. We provide conditions for zero duality gap in conjugate duality. For Lagrange duality, intersection property is applied to obtain zero duality gap. Connection between Lagrange dual and conjugate dual is also established. Examples related to convex and weakly convex functions are given.
In a recent paper it has been shown that if Cesaro and Abel limits for a certain discrete time optimal control problem are not equal, then there is a duality gap between a certain infinite-dimensional linear programming problem and its dual. In this paper we construct an example of a problem satisfying the assumptions of the aforementioned paper, where Cesaro and Abel limits are different.
Fourier-Motzkin elimination is a projection algorithm for solving finite linear programs. We extend Fourier-Motzkin elimination to semi-infinite linear programs which are linear programs with finitely many variables and infinitely many constraints. Applying projection leads to new characterizations of important properties for primal-dual pairs of semi-infinite programs such as zero duality gap, feasibility, boundedness, and solvability. Extending the Fourier-Motzkin elimination procedure to semi-infinite linear programs yields a new classification of variables that is used to determine the exi…
We discuss a weak constraint qualification for conic linear programs and its applications for a few classes of cones. This constraint qualification is used to give a solution to a problem proposed by Shapiro and Zǎlinescu and show that if a closed convex cone is such that all its non-trivial faces are polyhedral and all the non-trivial exposed faces of its dual are polyhedral, then the duality gap is zero as long as the primal and dual problems are feasible. Moreover, the common optimal value must be attained at least at one of the sides. We also show an example of a cone that meets the requir…
Using tools provided by the theory of abstract convexity, we extend conditions for zero duality gap to the context of nonconvex and nonsmooth optimization. Mimicking the classical setting, an abstract convex function is the upper envelope of a family of abstract affine functions (being conventional vertical translations of the abstract linear functions). We establish new conditions for zero duality gap under no topological assumptions on the space of abstract linear functions. In particular, we prove that the zero duality gap property can be fully characterized in terms of an inclusion involvi…
We introduce and study a new dual condition which characterizes zero duality gap in nonsmooth convex optimization. We prove that our condition is weaker than all existing constraint qualifications, including the closed epigraph condition. Our dual condition was inspired by, and is weaker than, the so-called Bertsekas' condition for monotropic programming problems. We give several corollaries of our result and special cases as applications. We pay special attention to the polyhedral and sublinear cases, and their implications in convex optimization.
The paper deals with the optimal control problem described by second order evolution differential inclusions; to this end first we use an auxiliary problem with second order discrete and discrete-approximate inclusions. Then applying infimal convolution concept of convex functions, step by step we construct the dual problems for discrete, discrete-approximate and differential inclusions and prove duality results. It seems that the Euler-Lagrange type inclusions are "duality relations" for both primary and dual problems and that the dual problem for discrete-approximate problem make a bridge be…
We present a novel analysis of semidefinite programs (SDPs) with positive duality gaps, i.e. different optimal values in the primal and dual problems. These SDPs are extremely pathological, often unsolvable, and also serve as models of more general pathological convex programs. However, despite their allure, they are not well understood even when they have just two variables. We first completely characterize two variable SDPs with positive gaps; in particular, we transform them into a standard form that makes the positive gap trivial to recognize. The transformation is very simple, as it mostl…
In this paper we introduce a new dual program, which is representable as a semi-definite linear programming problem, for a primal convex minimax programming model problem and show that there is no duality gap between the primal and the dual whenever the functions involved are SOS-convex polynomials. Under a suitable constraint qualification, we derive strong duality results for this class of minimax problems. Consequently, we present applications of our results to robust SOS-convex programming problems under data uncertainty and to minimax fractional programming problems with SOS-convex polyno…
Geometric duality theory for multiple objective linear programming problems turned out to be very useful for the development of efficient algorithms to generate or approximate the whole set of nondominated points in the outcome space. This article extends the geometric duality theory to convex vector optimization problems.
Mixed integer quadratic programming (MIQP) is the problem of minimizing a convex quadratic function over mixed integer points in a rational polyhedron. This paper focuses on the augmented Lagrangian dual (ALD) for MIQP. ALD augments the usual Lagrangian dual with a weighted nonlinear penalty on the dualized constraints. We first prove that ALD will reach a zero duality gap asymptotically as the weight on the penalty goes to infinity under some mild conditions on the penalty function. We next show that a finite penalty weight is enough for a zero gap when we use any norm as the penalty function…
We consider the problem of minimizing a sum of non-convex functions over a compact domain, subject to linear inequality and equality constraints. Approximate solutions can be found by solving a convexified version of the problem, in which each function in the objective is replaced by its convex envelope. We propose a randomized algorithm to solve the convexified problem which finds an $ε$-suboptimal solution to the original problem. With probability one, $ε$ is bounded by a term proportional to the maximal number of active constraints in the problem. The bound does not depend on the number of …
We investigate Lagrangian duality for nonconvex optimization problems. To this aim we use the $Φ$-convexity theory and minimax theorem for $Φ$-convex functions. We provide conditions for zero duality gap and strong duality. Among the classes of functions, to which our duality results can be applied, are prox-bounded functions, DC functions, weakly convex functions and paraconvex functions.
We consider semi-infinite linear programs with countably many constraints indexed by the natural numbers. When the constraint space is the vector space of all real valued sequences, we show the finite support (Haar) dual is equivalent to the algebraic Lagrangian dual of the linear program. This settles a question left open by Anderson and Nash [Linear programming in infinite dimensional spaces : theory and applications, Wiley 1987]. This result implies that if there is a duality gap between the primal linear program and its finite support dual, then this duality gap cannot be closed by conside…
This paper is concerned with the study of constrained statistical learning problems, the unconstrained version of which are at the core of virtually all of modern information processing. Accounting for constraints, however, is paramount to incorporate prior knowledge and impose desired structural and statistical properties on the solutions. Still, solving constrained statistical problems remains challenging and guarantees scarce, leaving them to be tackled using regularized formulations. Though practical and effective, selecting regularization parameters so as to satisfy requirements is challe…
In this article we develop a duality principle suitable for a large class of problems in optimization. The main result is obtained through basic tools of convex analysis and duality theory. We establish a correct relation between the critical points of the primal and dual formulations and formally prove there is no duality gap between such formulations, in a local extremal context.
This paper deals with the problem of linear programming with inexact data represented by real closed intervals. Optimization problems with interval data arise in practical computations and they are of theoretical interest for more than forty years. We extend the concept of duality gap (DG), the difference between the primal and its dual optimal value, into interval linear programming. We consider two situations: First, DG is zero for every realization of interval parameters (the so called strongly zero DG) and, second, DG is zero for at least one realization of interval parameters (the so call…
Screening rules allow to early discard irrelevant variables from the optimization in Lasso problems, or its derivatives, making solvers faster. In this paper, we propose new versions of the so-called $\textit{safe rules}$ for the Lasso. Based on duality gap considerations, our new rules create safe test regions whose diameters converge to zero, provided that one relies on a converging solver. This property helps screening out more variables, for a wider range of regularization parameter values. In addition to faster convergence, we prove that we correctly identify the active sets (supports) of…
This work addresses the Robust counterpart of the Shortest Path Problem (RSPP) with a correlated uncertainty set. Since this problem is hard, a heuristic approach, based on Frank-Wolfe's algorithm named Discrete Frank-Wolf (DFW), has recently been proposed. The aim of this paper is to propose a semi-definite programming relaxation for the RSPP that provides a lower bound to validate approaches such as DFW Algorithm. The relaxed problem results from a bidualization that is done {through} a reformulation of the RSPP into a quadratic problem. Then the relaxed problem is solved using a sparse vers…
This article develops a duality principle applicable to a large class of variational problems. Firstly, we apply the results to a Ginzburg-Landau type model. In a second step, we develop another duality principle and related primal dual variational formulation and such an approach includes optimality conditions which guarantee zero duality gap between the primal and dual formulations. We emphasize in both cases the dual variational formulations obtained have large regions of concavity about the critical points in question.
We present a new duality theory for non-convex variational problems, under possibly mixed Dirichlet and Neumann boundary conditions. The dual problem reads nicely as a linear programming problem, and our main result states that there is no duality gap. Further, we provide necessary and sufficient optimality conditions, and we show that our duality principle can be reformulated as a min-max result which is quite useful for numerical implementations. As an example, we illustrate the application of our method to a celebrated free boundary problem. The results were announced in \cite{BoFr}.
We associate with each convex optimization problem, posed on some locally convex space, with infinitely many constraints indexed by the set T, and a given non-empty family H of finite subsets of T, a suitable Lagrangian-Haar dual problem. We obtain necessary and sufficient conditions for H-reducibility, that is, equivalence to some subproblem obtained by replacing the whole index set T by some element of H. Special attention is addressed to linear optimization, infinite and semi-infinite, and to convex problems with a countable family of constraints. Results on zero H-duality gap and on H-(sta…
This article develops a duality principle for a class of optimization problems in $\mathbb{R}^n$. The results are obtained based on standard tools of convex analysis and on a well known result of Toland for D.C. optimization. Global sufficient optimality conditions are also presented as well as relations between the critical points of the primal and dual formulations. Finally we formally prove there is no duality gap between the primal and dual formulations in a local extremal context.
By applying the perturbation function approach, we propose the Lagrangian and the conjugate duals for minimization problems of the sum of two, generally nonconvex, functions. The main tools are the $Φ$-convexity theory and minimax theorems for $Φ$-convex functions. We provide conditions ensuring zero duality gap and introduce $Φ$-Karush-Kuhn-Tucker conditions that characterize solutions to primal and dual problems. We also discuss the relationship between the dual problems introduced in the present investigation and some conjugate-type duals existing in the literature.
This article develops duality principles applicable to non-convex models in the calculus of variations. The results here developed are applied to Ginzburg-Landau type equations. For the first and second duality principles, through an optimality criterion developed for the dual formulations, we qualitatively classify the critical points of the primal and dual functionals in question. We formally prove there is no duality gap between the primal and dual formulations in a local extremal context. Finally, in the last sections, we present a global existence result, a duality principle and respectiv…
We present simple compact proofs of the strong and weak duality theorems of tropical linear programming. It follows that there is no duality gap for a pair of tropical primal-dual problems. This result together with known properties of subeigenvectors enables us to directly solve a special tropical linear program with two-sided constraints. We also study the duality gap in tropical integer linear programming. A direct solution is available for the primal problem. An algorithm of quadratic complexity is presented for the dual problem. A direct solution is available provided that all coefficient…
This paper proposes a general duality framework for the problem of minimizing a convex integral functional over a space of stochastic processes adapted to a given filtration. The framework unifies many well-known duality frameworks from operations research and mathematical finance. The unification allows the extension of some useful techniques from these two fields to a much wider class of problems. In particular, combining certain finite-dimensional techniques from convex analysis with measure theoretic techniques from mathematical finance, we are able to close the duality gap in some situati…
Despite the vast literature on DRS and ADMM, there has been very little work analyzing their behavior under pathologies. Most analyses assume a primal solution exists, a dual solution exists, and strong duality holds. When these assumptions are not met, i.e., under pathologies, the theory often breaks down and the empirical performance may degrade significantly. In this paper, we establish that DRS only requires strong duality to work, in the sense that asymptotically iterates are approximately feasible and approximately optimal.
Autonomous agents must often deal with conflicting requirements, such as completing tasks using the least amount of time/energy, learning multiple tasks, or dealing with multiple opponents. In the context of reinforcement learning~(RL), these problems are addressed by (i)~designing a reward function that simultaneously describes all requirements or (ii)~combining modular value functions that encode them individually. Though effective, these methods have critical downsides. Designing good reward functions that balance different objectives is challenging, especially as the number of objectives g…
We show that sparsity constrained optimization problems over low dimensional spaces tend to have a small duality gap. We use the Shapley-Folkman theorem to derive both data-driven bounds on the duality gap, and an efficient primalization procedure to recover feasible points satisfying these bounds. These error bounds are proportional to the rate of growth of the objective with the target cardinality, which means in particular that the relaxation is nearly tight as soon as the target cardinality is large enough so that only uninformative features are added.
This paper studies duality and optimality conditions in general convex stochastic optimization problems introduced by Rockafellar and Wets in 1976. We derive an explicit dual problem in terms of two dual variables, one of which is the shadow price of information while the other one gives the marginal cost of a perturbation much like in classical Lagrangian duality. Existence of primal solutions and the absence of duality gap are obtained without compactness or boundedness assumptions. In the context of financial mathematics, the relaxed assumptions are satisfied under the well-known no-arbitra…
This article develops a new primal dual formulation for the Kirchhoff-Love non-linear plate model. At first we establish a duality principle which includes sufficient conditions of global optimality through the dual formulation. At this point we highlight this first duality principle is specially suitable for the case in which the membrane stress tensor is negative definite. In a second step, from such a general principle, we develop a primal dual variational formulation which also includes the corresponding sufficient conditions for global optimality. The results are based on standard tools o…
In spite of the growing consideration for optimal execution in the financial mathematics literature, numerical approximations of optimal trading curves are almost never discussed. In this article, we present a numerical method to approximate the optimal strategy of a trader willing to unwind a large portfolio. The method we propose is very general as it can be applied to multi-asset portfolios with any form of execution costs, including a bid-ask spread component, even when participation constraints are imposed. Our method, based on convex duality, only requires Hamiltonian functions to have $…
Georgia Institute of Technology（佐治亚理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Politecnico di Torino（都灵理工大学）
State-of-the-art techniques for simultaneous localization and mapping (SLAM) employ iterative nonlinear optimization methods to compute an estimate for robot poses. While these techniques often work well in practice, they do not provide guarantees on the quality of the estimate. This paper shows that Lagrangian duality is a powerful tool to assess the quality of a given candidate solution. Our contribution is threefold. First, we discuss a revised formulation of the SLAM inference problem. We show that this formulation is probabilistically grounded and has the advantage of leading to an optimi…
