In this paper, we propose a distributed algorithm for solving large-scale separable convex problems using Lagrangian dual decomposition and the interior-point framework. By adding self-concordant barrier terms to the ordinary Lagrangian, we prove under mild assumptions that the corresponding family of augmented dual functions is self-concordant. This makes it possible to efficiently use the Newton method for tracing the central path. We show that the new algorithm is globally convergent and highly parallelizable and thus it is suitable for solving large-scale separable convex problems.
The augmented Lagrangian method (ALM) is classic for canonical convex programming problems with linear constraints, and it finds many applications in various scientific computing areas. A major advantage of the ALM is that the step for updating the dual variable can be further relaxed with a step size in $(0,2)$, and this advantage can easily lead to numerical acceleration for the ALM. When a separable convex programming problem is discussed and a corresponding splitting version of the classic ALM is considered, convergence may not be guaranteed and thus it is seemingly impossible that a step …
We develop a new inexact interior-point Lagrangian decomposition method to solve a wide range class of constrained composite convex optimization problems. Our method relies on four techniques: Lagrangian dual decomposition, self-concordant barrier smoothing, path-following, and proximal-Newton technique. It also allows one to approximately compute the solution of the primal subproblems (called the slave problems), which leads to inexact oracles (i.e., inexact gradients and Hessians) of the smoothed dual problem (called the master problem). The smoothed dual problem is nonsmooth, we propose to …
In this paper, we propose an inexact perturbed path-following algorithm in the framework of Lagrangian dual decomposition for solving large-scale structured convex optimization problems. Unlike the exact versions considered in literature, we allow one to solve the primal problem inexactly up to a given accuracy. The inexact perturbed algorithm allows to use both approximate Hessian matrices and approximate gradient vectors to compute Newton-type directions for the dual problem. The algorithm is divided into two phases. The first phase computes an initial point which makes use of inexact pertur…
In this paper we propose a new inexact dual decomposition algorithm for solving separable convex optimization problems. This algorithm is a combination of three techniques: dual Lagrangian decomposition, smoothing and excessive gap. The algorithm requires only one primal step and two dual steps at each iteration and allows one to solve the subproblem of each component inexactly and in parallel. Moreover, the algorithmic parameters are updated automatically without any tuning strategy as in augmented Lagrangian approaches. We analyze the convergence of the algorithm and estimate its $O(\frac{1}…
We consider the convex minimization model with both linear equality and inequality constraints, and reshape the classic augmented Lagrangian method (ALM) by balancing its subproblems. As a result, one of its subproblems decouples the objective function and the coefficient matrix without any extra condition, and the other subproblem becomes a positive definite system of linear equations or a positive definite linear complementary problem. The balanced ALM advances the classic ALM by enlarging its applicable range, balancing its subproblems, and improving its implementation. We also extend our d…
Combining Lagrangian Decomposition and Excessive Gap Smoothing Technique for Solving Large-Scale Separable Convex Optimization Problems
Department of Electrical Engineering (ESAT-SCD) and Optimization in Engineering Center (OPTEC), K.U. Leuven（荷语鲁汶大学电气工程系（ESAT-SCD）与工程优化中心（OPTEC））；Hanoi University of Science（河内科技大学）
A new decomposition optimization algorithm, called \textit{path-following gradient-based decomposition}, is proposed to solve separable convex optimization problems. Unlike path-following Newton methods considered in the literature, this algorithm does not requires any smoothness assumption on the objective function. This allows us to handle more general classes of problems arising in many real applications than in the path-following Newton methods. The new algorithm is a combination of three techniques, namely smoothing, Lagrangian decomposition and path-following gradient framework. The algo…
Decomposition techniques for linear programming are difficult to extend to conic optimization problems with general non-polyhedral convex cones because the conic inequalities introduce an additional nonlinear coupling between the variables. However in many applications the convex cones have a partially separable structure that allows them to be characterized in terms of simpler lower-dimensional cones. The most important example is sparse semidefinite programming with a chordal sparsity pattern. Here partial separability derives from the clique decomposition theorems that characterize positive…
The aim of this manuscript is to approach by means of first order differential equations/inclusions convex programming problems with two-block separable linear constraints and objectives, whereby (at least) one of the components of the latter is assumed to be strongly convex. Each block of the objective contains a further smooth convex function. We investigate the dynamical system proposed and prove that its trajectories asymptotically converge to a saddle point of the Lagrangian of the convex optimization problem. Time discretization of the dynamical system leads to the alternating minimizati…
Several well known large scale linear programming decomposition methodologies exist. Benders Decomposition, which covers the case where some small subset of variables link the otherwise separable subproblems. Dantzig-Wolfe decomposition and Lagrangian decompositions, which cover the case where some few constraints link the otherwise separable subproblems, and finally the "Cross-Decomposition" originating from TJ Van Roy which enables one to deal with both linking constraints and linking variables by essentially alternating iteratively between the Benders and the Lagrangian Decomposition. In th…
We consider a nonsmooth optimization problem on Riemannian manifold, whose objective function is the sum of a differentiable component and a nonsmooth convex function. We propose a manifold inexact augmented Lagrangian method (MIALM) for the considered problem. The problem is reformulated to a separable form. By utilizing the Moreau envelope, we get a smoothing subproblem at each iteration of the proposed method. Theoretically, under suitable assumptions, the convergence to critical point of the proposed method is established. In particular, under the condition of that the approximate global m…
Powerful interior-point methods (IPM) based commercial solvers, such as Gurobi and Mosek, have been hugely successful in solving large-scale linear programming (LP) problems. The high efficiency of these solvers depends critically on the sparsity of the problem data and advanced matrix factorization techniques. For a large scale LP problem with data matrix $A$ that is dense (possibly structured) or whose corresponding normal matrix $AA^T$ has a dense Cholesky factor (even with re-ordering), these solvers may require excessive computational cost and/or extremely heavy memory usage in each inter…
This paper considers large scale constrained convex (possibly composite and non-separable) programs, which are usually difficult to solve by interior point methods or other Newton-type methods due to the non-smoothness or the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, they are often solved by first order gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, also known as the Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with an $O(1/ε^2)$ convergence time. Recently, a new …
A multi-agent optimization problem motivated by the management of energy systems is discussed. The associated cost function is separable and convex although not necessarily strongly convex and there exist edge-based coupling equality constraints. In this regard, we propose a distributed algorithm based on solving the dual of the augmented problem. Furthermore, we consider that the communication network might be time-varying and the algorithm might be carried out asynchronously. The time-varying nature and the asynchronicity are modeled as random processes. Then, we show the convergence and the…
Regularization and interior point approaches offer valuable perspectives to address constrained nonlinear optimization problems in view of control applications. This paper discusses the interactions between these techniques and proposes an algorithm that synergistically combines them. Building a sequence of closely related subproblems and approximately solving each of them, this approach inherently exploits warm-starting, early termination, and the possibility to adopt subsolvers tailored to specific problem structures. Moreover, by relaxing the equality constraints with a proximal penalty, th…
Dual decomposition is a powerful technique for deriving decomposition schemes for convex optimization problems with separable structure. Although the Augmented Lagrangian is computationally more stable than the ordinary Lagrangian, the prox-term destroys the separability of the given problem. In this paper we use another approach to obtain a smooth Lagrangian, based on a smoothing technique developed by Nesterov, which preserves separability of the problem. With this approach we derive a new decomposition method, called proximal center algorithm, which from the viewpoint of efficiency estimate…
The auxiliary problem principle of augmented Lagrangian (APP-AL), proposed by Cohen and Zhu (1984), aims to find the solution of a constrained optimization problem through a sequence of auxiliary problems involving augmented Lagrangian. The merits of this approach are two folds. First, the core function is usually separable, which makes the subproblems at each step decomposable and particularly attractive for parallel computing. Second, the choice of the core function is quite flexible. Consequently, by carefully specifying this function, APP-AL may reduce to some standard optimization algorit…
We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hetenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality cons…
We consider non-autonomous calculus of variations problems with a state constraint represented by a given closed set. We prove that if the interior of the Clarke tangent cone of the state constraint set is non-empty (this is the constraint qualification that we suggest here), then the necessary optimality conditions apply in the normal form. We establish normality results for (weak) local minimizers and global minimizers, employing two different approaches and invoking slightly diverse assumptions. More precisely, for the local minimizers result, the Lagrangian is supposed to be Lipschitz with…
With the help of a logarithmic barrier augmented Lagrangian function, we can obtain closed-form solutions of slack variables of logarithmic-barrier problems of nonlinear programs. As a result, a two-parameter primal-dual nonlinear system is proposed, which corresponds to the Karush-Kuhn-Tucker point and the infeasible stationary point of nonlinear programs, respectively, as one of two parameters vanishes. Based on this distinctive system, we present a primal-dual interior-point method capable of rapidly detecting infeasibility of nonlinear programs. The method generates interior-point iterates…
We study deterministic and stochastic primal-dual sub-gradient algorithms for distributed optimization of a separable objective function with global inequality constraints. In both algorithms, the norm of the Lagrangian multipliers are controlled by augmenting the corresponding Lagrangian function with a quadratic regularization term. Specifically, we show that when the stepsize of each algorithm satisfies a certain restriction, the norm of the Lagrangian multipliers is upper bounded by an expression that is inversely proportional to the parameter of the regularization. We use this result to c…
This paper considers large scale constrained convex programs, which are usually not solvable by interior point methods or other Newton-type methods due to the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, large scale constrained convex programs are often solved by gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, aka, Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with the $O(1/\sqrt{t})$ convergence rate, where $t$ is the number of iterations. If the objective and constra…
In this study, calculations necessary to solve the large scale linear programming problems in two operating systems, Linux and Windows 7 (Win), are compared using two different methods. Relying on the interior-point methods, linear-programming interior point solvers (LIPSOL) software was used for the first method and relying on an augmented Lagrangian method-based algorithm, the second method used the generalized derivative. The performed calculations for various problems show the produced random in the Linux operating system (OS) and Win OS indicate the efficiency of the performed calculation…
We develop two new variants of alternating direction methods of multipliers (ADMM) and two parallel primal-dual decomposition algorithms to solve a wide range class of constrained convex optimization problems. Our approach relies on a novel combination of the augmented Lagrangian framework, partial alternating/linearization scheme, Nesterov's acceleration technique, and adaptive strategy. The proposed algorithms have the following new features compared to existing ADMM variants. First, they have a Nesterov's acceleration step on the primal variables instead of the dual ones as in several ADMM …
A primal-dual interior point trust region method for second-order stationary points of Riemannian inequality-constrained optimization problems
Graduate School of Information Science and Technology, The University of Tokyo（东京大学大学院信息理工学系研究科）；Center for Advanced Intelligence Project, RIKEN（理化学研究所先进智能项目中心）；Faculty of Science and Technology, Seikei University（立教大学理学部）
Although much research has been devoted to the problem of restoring Poissonian images, namely in the fields of medical and astronomical imaging, applying the state of the art regularizers (such as those based on wavelets or total variation) to this class of images is still an open research front. This paper proposes a new image deconvolution approach for images with Poisson statistical models, with the following building blocks: (a) a standard regularization/MAP criterion, combining the Poisson log-likelihood with a regularizer (log-prior) is adopted; (b) the resulting optimization problem (wh…
This paper considers decentralized consensus optimization problems where different summands of a global objective function are available at nodes of a network that can communicate with neighbors only. The proximal method of multipliers is considered as a powerful tool that relies on proximal primal descent and dual ascent updates on a suitably defined augmented Lagrangian. The structure of the augmented Lagrangian makes this problem non-decomposable, which precludes distributed implementations. This problem is regularly addressed by the use of the alternating direction method of multipliers. T…
The augmented Lagrangian (AL) method that solves convex optimization problems with linear constraints has drawn more attention recently in imaging applications due to its decomposable structure for composite cost functions and empirical fast convergence rate under weak conditions. However, for problems such as X-ray computed tomography (CT) image reconstruction and large-scale sparse regression with "big data", where there is no efficient way to solve the inner least-squares problem, the AL method can be slow due to the inevitable iterative inner updates. In this paper, we focus on solving reg…
