Computational methods are proposed for solving a convex quadratic program (QP). Active-set methods are defined for a particular primal and dual formulation of a QP with general equality constraints and simple lower bounds on the variables. In the first part of the paper, two methods are proposed, one primal and one dual. These methods generate a sequence of iterates that are feasible with respect to the equality constraints associated with the optimality conditions of the primal-dual form. The primal method maintains feasibility of the primal inequalities while driving the infeasibilities of the dual inequalities to zero. The dual method maintains feasibility of the dual inequalities while moving to satisfy the primal inequalities. In each of these methods, the search directions satisfy a KKT system of equations formed from Hessian and constraint components associated with an appropriate column basis. The composition of the basis is specified by an active-set strategy that guarantees the nonsingularity of each set of KKT equations. Each of the proposed methods is a conventional active-set method in the sense that an initial primal- or dual-feasible point is required. In the second part of the paper, it is shown how the quadratic program may be solved as a coupled pair of primal and dual quadratic programs created from the original by simultaneously shifting the simple-bound constraints and adding a penalty term to the objective function. Any conventional column basis may be made optimal for such a primal-dual pair of shifted-penalized problems. The shifts are then updated using the solution of either the primal or the dual shifted problem. An obvious application of this approach is to solve a shifted dual QP to define an initial feasible point for the primal (or vice versa). The computational performance of each of the proposed methods is evaluated on a set of convex problems.
We propose primal-dual stochastic mirror descent for the convex optimization problems with functional constraints. We obtain the rate of convergence in terms of probability of large deviations.
Linear constrained convex programming has many practical applications, including support vector machine and machine learning portfolio problems. We propose the randomized primal-dual coordinate (RPDC) method, a randomized coordinate extension of the first-order primal-dual method by Cohen and Zhu, 1984 and Zhao and Zhu, 2019, to solve linear constrained convex programming. We randomly choose a block of variables based on a uniform distribution, linearize, and apply a Bregman-like function (core function) to the selected block to obtain simple parallel primal-dual decomposition. We then establi…
Most recently, He and Yuan [arXiv:2108.08554, 2021] have proposed a balanced augmented Lagrangian method (ALM) for the canonical convex programming problem with linear constraints, which advances the original ALM by balancing its subproblems and improving its implementation. In this short note, we propose a dual-primal version of the balanced ALM, which updates the new iterate via a conversely dual-primal iterative order formally. The proposed method inherits all advantages of the prototype balanced ALM, and its convergence analysis can be well conducted in the context of variational inequalit…
This paper considers an inexact primal-dual algorithm for semi-infinite programming (SIP) for which it provides general error bounds. To implement the dual variable update, we create a new prox function for nonnegative measures which turns out to be a generalization of the Kullback-Leibler divergence for probability distributions. We show that under suitable conditions on the error, this algorithm achieves an $\mathcal{O}(1/\sqrt{K})$ rate of convergence in terms of the optimality gap and constraint violation. We then use our general error bounds to analyze the convergence and sample complexit…
We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems. Both for primal and dual oracles, the proposed methods are optimal in terms of the number of communication steps. However, for all classes of the objective, the optimality in terms of the number of oracle calls per node takes place only up to a logarithmic factor and the notion of smoothness. By using mini-batching technique, we show that the proposed methods with stochastic oracle can be additionally parallelized at each node. The considered algorithms can be applied to many data sc…
This paper mainly concerns with the primal superlinear convergence of the quasi-Newton sequential quadratic programming (SQP) method for piecewise linear-quadratic composite optimization problems. We show that the latter primal superlinear convergence can be justified under the noncriticality of Lagrange multipliers and a version of the Dennis-More condition. Furthermore, we show that if we replace the noncriticality condition with the second-order sufficient condition, this primal superlinear convergence is equivalent with an appropriate version of the Dennis-More condition. We also recover B…
This study addresses primal-dual dynamics for a stochastic programming problem for capacity network design. It is proven that consensus can be achieved on the \textit{here and now} variables which represent the capacity of the network. The main contribution is a heuristic approach which involves the formulation of the problem as a mean-field game. Every agent in the mean-field game has control over its own primal-dual dynamics and seeks consensus with neighboring agents according to a communication topology. We obtain theoretical results concerning the existence of a mean-field equilibrium. Mo…
Since more than three decades, interior-point methods proved very useful for optimization, from linear over semidefinite to conic (and partly beyond non-convex) programming; despite the fact that already in the semidefinite case (even when strong duality holds) "hard" problems are known. We shade a light on a rather surprising restriction in the non-commutative world (of semidefinite programming), namely "commutative" paths and propose a new family of solvers that is able to use the full richness of "non-commutative" search directions: (primal) feasible-interior-point methods. Beside a detaile…
We propose a semi-proximal augmented Lagrangian based decomposition method for convex composite quadratic conic programming problems with primal block angular structures. Using our algorithmic framework, we are able to naturally derive several well known augmented Lagrangian based decomposition methods for stochastic programming such as the diagonal quadratic approximation method of Mulvey and Ruszczyński. Moreover, we are able to derive novel enhancements and generalizations of these well known methods. We also propose a semi-proximal symmetric Gauss-Seidel based alternating direction method …
In this work we study a constrained monotone inclusion involving the normal cone to a closed vector subspace and a priori information on primal solutions. We model this information by imposing that solutions belongs to the fixed point set of an averaged nonexpansive mapping. We characterize the solutions using an auxiliary inclusion that involves the partial inverse operator. Then, we propose the primal-dual partial inverse splitting and we prove its weak convergence to a solution of the inclusion, generalizing several methods in the literature. The efficiency of the proposed method is illustr…
In this paper we provide a detailed analysis of the iteration complexity of dual first order methods for solving conic convex problems. When it is difficult to project on the primal feasible set described by convex constraints, we use the Lagrangian relaxation to handle the complicated constraints and then, we apply dual first order algorithms for solving the corresponding dual problem. We give convergence analysis for dual first order algorithms (dual gradient and fast gradient algorithms): we provide sublinear or linear estimates on the primal suboptimality and feasibility violation of the g…
Interior Point Methods are widely used to solve Linear Programming problems. In this work, we present two primal affine scaling algorithms to achieve faster convergence in solving Linear Programming problems. In the first algorithm, we integrate Nesterov's restarting strategy in the primal affine scaling method with an extra parameter, which in turn generalizes the original primal affine scaling method. We provide the proof of convergence for the proposed generalized algorithm considering long step size. We also provide the proof of convergence for the primal and dual sequence without the dege…
Nonlinear Convex Cone Programming (NCCP) problems are important and have many practical applications. In this paper, we introduces a flexible first-order primal-dual algorithm called the Variant Auxiliary Problem Principle (VAPP) for solving NCCP problems when the objective function and constraints are smooth and may be nonsmooth. Each iteration of VAPP generates a nonlinear approximation to the primal problem of an augmented Lagrangian method. The approximation incorporates both linearization and a variable distance-like function, and then the iterations of VAPP provide one decomposition prop…
We propose an early termination technique for mixed integer conic programming for use within branch-and-bound based solvers. Our approach generalizes previous early termination results for ADMM-based solvers to a broader class of primal-dual algorithms, including both operator splitting methods and interior point methods. The complexity for checking early termination is $O(n)$ for each termination check assuming a bounded problem domain. We show that this domain restriction can be relaxed for problems whose data satisfies a simple rank condition, in which case each check requires an $O(n^2)$ s…
This paper presents the Distributed Primal Outer Approximation (DiPOA) algorithm for solving Sparse Convex Programming (SCP) problems with separable structures, efficiently, and in a decentralized manner. The DiPOA algorithm development consists of embedding the recently proposed Relaxed Hybrid Alternating Direction Method of Multipliers (RH-ADMM) algorithm into the Outer Approximation (OA) algorithm. We also propose two main improvements to control the quality and the number of cutting planes that approximate nonlinear functions. In particular, the RH-ADMM algorithm acts as a distributed nume…
In this chapter we derive computational complexity certifications of first order inexact dual methods for solving general smooth constrained convex problems which can arise in real-time applications, such as model predictive control. When it is difficult to project on the primal constraint set described by a collection of general convex functions, we use the Lagrangian relaxation to handle the complicated constraints and then, we apply dual (fast) gradient algorithms based on inexact dual gradient information for solving the corresponding dual problem. The iteration complexity analysis is base…
In this paper we present the solver DuQuad specialized for solving general convex quadratic problems arising in many engineering applications. When it is difficult to project on the primal feasible set, we use the (augmented) Lagrangian relaxation to handle the complicated constraints and then, we apply dual first order algorithms based on inexact dual gradient information for solving the corresponding dual problem. The iteration complexity analysis is based on two types of approximate primal solutions: the primal last iterate and an average of primal iterates. We provide computational complex…
We propose and analyse primal-dual interior-point algorithms for convex optimization problems in conic form. The families of algorithms we analyse are so-called short-step algorithms and they match the current best iteration complexity bounds for primal-dual symmetric interior-point algorithm of Nesterov and Todd, for symmetric cone programming problems with given self-scaled barriers. Our results apply to any self-concordant barrier for any convex cone. We also prove that certain specializations of our algorithms to hyperbolic cone programming problems (which lie strictly between symmetric co…
Classical primal-dual affine programming takes place over finite dimensional real vector spaces. This results in beautiful duality theory, connecting the optimal solu- tions of the primal maximization problem and the dual minimization problems. These results include the Existence Duality Theorem, which guarantees optimal solutions to any feasible bounded program; and the Strong Duality Theorem, which implies that optimal solutions for primal and dual programs must have the same objective value. In a common extension of classical affine programming, we see that the Strong Duality does not hold …
Quasi-Newton methods are well known techniques for large-scale numerical optimization. They use an approximation of the Hessian in optimization problems or the Jacobian in system of nonlinear equations. In the Interior Point context, quasi-Newton algorithms compute low-rank updates of the matrix associated with the Newton systems, instead of computing it from scratch at every iteration. In this work, we show that a simplified quasi-Newton primal-dual interior point algorithm for linear programming enjoys polynomial worst-case iteration complexity. Feasible and infeasible cases of the algorithm…
We present PDLP, a practical first-order method for linear programming (LP) that can solve to the high levels of accuracy that are expected in traditional LP applications. In addition, it can scale to very large problems because its core operation is matrix-vector multiplications. PDLP is derived by applying the primal-dual hybrid gradient (PDHG) method, popularized by Chambolle and Pock (2011), to a saddle-point formulation of LP. PDLP enhances PDHG for LP by combining several new techniques with older tricks from the literature; the enhancements include diagonal preconditioning, presolving, …
We study the problem of detecting infeasibility of large-scale linear programming problems using the primal-dual hybrid gradient method (PDHG) of Chambolle and Pock (2011). The literature on PDHG has mostly focused on settings where the problem at hand is assumed to be feasible. When the problem is not feasible, the iterates of the algorithm do not converge. In this scenario, we show that the iterates diverge at a controlled rate towards a well-defined ray. The direction of this ray is known as the infimal displacement vector $v$. The first contribution of our work is to prove that this vector…
Solving large scale convex semidefinite programming (SDP) problems has long been a challenging task numerically. Fortunately, several powerful solvers including SDPNAL, SDPNAL+ and QSDPNAL have recently been developed to solve linear and convex quadratic SDP problems to high accuracy successfully. These solvers are based on the augmented Lagrangian method (ALM) applied to the dual problems with the subproblems being solved by semismooth Newton-CG methods. Noticeably, thanks to Rockafellar's general theory on the proximal point algorithms, the primal iteration sequence generated by the ALM enjo…
We study the equivalence of several well-known sufficient optimality conditions for a general quadratically constrained quadratic program (QCQP). The conditions are classified in two categories. The first one is for determining an optimal solution and the second one is for finding an optimal value. The first category of conditions includes the existence of a saddle point of the Lagrangian function and the existence of a rank-1 optimal solution of the primal SDP relaxation of QCQP. The second category includes $η_p = ζ$, $η_d = ζ$, and $φ= ζ$, where $ζ$, $η_p$, $η_d$, and $φ$ denote the optimal…
This paper introduces the proximally stabilized Fischer-Burmeister method (FBstab); a new algorithm for convex quadratic programming that synergistically combines the proximal point algorithm with a primal-dual semismooth Newton-type method. FBstab is numerically robust, easy to warmstart, handles degenerate primal-dual solutions, detects infeasibility/unboundedness and requires only that the Hessian matrix be positive semidefinite. We outline the algorithm, provide convergence and convergence rate proofs, report some numerical results from model predictive control benchmarks, and also include…
In this work we consider a possibility to use the conception of $(δ, L)$-model of a function for optimization tasks, whereby solving a primal problem there is a necessity to recover a solution of a dual problem. The conception of $(δ, L)$-model is based on the conception of $(δ, L)$-oracle which was proposed by Devolder-Glineur-Nesterov, herewith the authors proposed approximate a function with an upper bound using a convex quadratic function with some additive noise $δ$. They managed to get convex quadratic upper bounds with noise even for nonsmooth functions. The conception of $(δ, L)$-model…
This paper investigates the input-output performance of secondary frequency controllers through the control-theoretic notion of $H_2$ norms. We consider a quadratic objective accounting for the cost of reserve procurement and provide exact analytical formulae for the performance of continuous-time aggregated averaging controllers. Then, we contrast it with distributed averaging controllers -- seeking optimality conditions such as identical marginal costs -- and primal-dual controllers which have gained attention as systematic techniques to design distributed algorithms solving convex optimizat…
Stochastic gradient methods (SGMs) have been widely used for solving stochastic optimization problems. A majority of existing works assume no constraints or easy-to-project constraints. In this paper, we consider convex stochastic optimization problems with expectation constraints. For these problems, it is often extremely expensive to perform projection onto the feasible set. Several SGMs in the literature can be applied to solve the expectation-constrained stochastic problems. We propose a novel primal-dual type SGM based on the Lagrangian function. Different from existing methods, our metho…
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
In this article, we consider the primal-dual path-following method and the trust-region updating strategy for the standard linear programming problem. For the rank-deficient problem with the small noisy data, we also give the preprocessing method based on the QR decomposition with column pivoting. Then, we prove the global convergence of the new method when the initial point is strictly primal-dual feasible. Finally, for some rank-deficient problems with or without the small noisy data from the NETLIB collection, we compare it with other two popular interior-point methods, i.e. the subroutine …
We study infeasible-start primal-dual interior-point methods for convex optimization problems given in a typically natural form we denote as Domain-Driven formulation. Our algorithms extend many advantages of primal-dual interior-point techniques available for conic formulations, such as the current best complexity bounds, and more robust certificates of approximate optimality, unboundedness, and infeasibility, to Domain-Driven formulations. The complexity results are new for the infeasible-start setup used, even in the case of linear programming. In addition to complexity results, our algorit…
This paper proposes TriPD, a new primal-dual algorithm for minimizing the sum of a Lipschitz-differentiable convex function and two possibly nonsmooth convex functions, one of which is composed with a linear mapping. We devise a randomized block-coordinate version of the algorithm which converges under the same stepsize conditions as the full algorithm. It is shown that both the original as well as the block-coordinate scheme feature linear convergence rate when the functions involved are either piecewise linear-quadratic, or when they satisfy a certain quadratic growth condition (which is wea…
In this paper, a projected primal-dual gradient flow of augmented Lagrangian is presented to solve convex optimization problems that are not necessarily strictly convex. The optimization variables are restricted by a convex set with computable projection operation on its tangent cone as well as equality constraints. As a supplement of the analysis in \cite{niederlander2016distributed}, we show that the projected dynamical system converges to one of the saddle points and hence finding an optimal solution. Moreover, the problem of distributedly maximizing the algebraic connectivity of an undirec…
Several well-known algorithms in the field of combinatorial optimization can be interpreted in terms of the primal-dual method for solving linear programs. For example, Dijkstra's algorithm, the Ford-Fulkerson algorithm, and the Hungarian algorithm can all be viewed as the primal-dual method applied to the linear programming formulations of their respective optimization problems. Roughly speaking, successfully applying the primal-dual method to an optimization problem that can be posed as a linear program relies on the ability to find a simple characterization of the optimal solutions to a rel…
Binary Integer Programming (BIP) problems are of interest due in part to the difficulty they pose and because of their various applications, including those in graph theory, combinatorial optimization and network optimization. In this note, we explicitly state the Lovasz-Schrijver Semidefinite Programming (SDP) relaxation (in primal-standard form) for a BIP problem, a relaxation that yields a tighter upper-bound than the canonical Linear Programming relaxation.
