We propose a distributed algorithm, named Distributed Alternating Direction Method of Multipliers (D-ADMM), for solving separable optimization problems in networks of interconnected nodes or agents. In a separable optimization problem there is a private cost function and a private constraint set at each node. The goal is to minimize the sum of all the cost functions, constraining the solution to be in the intersection of all the constraint sets. D-ADMM is proven to converge when the network is bipartite or when all the functions are strongly convex, although in practice, convergence is observed even when these conditions are not met. We use D-ADMM to solve the following problems from signal processing and control: average consensus, compressed sensing, and support vector machines. Our simulations show that D-ADMM requires less communications than state-of-the-art algorithms to achieve a given accuracy level. Algorithms with low communication requirements are important, for example, in sensor networks, where sensors are typically battery-operated and communicating is the most energy consuming operation.
An inexact accelerated stochastic Alternating Direction Method of Multipliers (AS-ADMM) scheme is developed for solving structured separable convex optimization problems with linear constraints. The objective function is the sum of a possibly nonsmooth convex function and a smooth function which is an average of many component convex functions. Problems having this structure often arise in machine learning and data mining applications. AS-ADMM combines the ideas of both ADMM and the stochastic gradient methods using variance reduction techniques. One of the ADMM subproblems employs a lineariza…
The alternating direction method of multipliers (ADMM) proposed by Glowinski and Marrocco is a benchmark algorithm for two-block separable convex optimization problems with linear equality constraints. It has been modified, specified, and generalized from various perspectives to tackle more concrete or complicated application problems. Despite its versatility and phenomenal popularity, it remains unknown whether or not the ADMM can be extended to separable convex optimization problems with linear inequality constraints. In this paper, we lay down the foundation of how to extend the ADMM to two…
The Alternating Direction Method of Multipliers (ADMM) has been proved to be effective for solving separable convex optimization subject to linear constraints. In this paper, we propose a Generalized Symmetric ADMM (GS-ADMM), which updates the Lagrange multiplier twice with suitable stepsizes, to solve the multi-block separable convex programming. This GS-ADMM partitions the data into two group variables so that one group consists of $p$ block variables while the other has $q$ block variables, where $p \ge 1$ and $q \ge 1$ are two integers. The two grouped variables are updated in a {\it Gauss…
In this paper, we propose and analyze an inexact version of the symmetric proximal alternating direction method of multipliers (ADMM) for solving linearly constrained optimization problems. Basically, the method allows its first subproblem to be solved inexactly in such way that a relative approximate criterion is satisfied. In terms of the iteration number $k$, we establish global $\mathcal{O} (1/ \sqrt{k})$ pointwise and $\mathcal{O} (1/ {k})$ ergodic convergence rates of the method for a domain of the acceleration parameters, which is consistent with the largest known one in the exact case.…
In a multi-agent network, we consider the problem of minimizing an objective function that is expressed as the sum of private convex and smooth functions, and a (possibly) non-differentiable convex regularizer. We propose a novel distributed second-order method based on the framework of Alternating Direction Method of Multipliers (ADMM), by invoking approximate Newton iterations to the primal update corresponding to the differentiable part. In order to achieve a distributed implementation, the total Hessian matrix is split into a diagonal component (locally computable) and an off-diagonal comp…
We consider the consensual distributed optimization problem and propose an asynchronous version of the Alternating Direction Method of Multipliers (ADMM) algorithm to solve it. The `asynchronous' part here refers to the fact that only one node/processor is updated (i.e. performs a minimization step) at each iteration of the algorithm. The selection of the node to be updated is decided by simulating a Markov chain. The proposed algorithm is shown to have a linear convergence property in expectation for the class of functions which are strongly convex and continuously differentiable.
As problems in machine learning, smartgrid dispatch, and IoT coordination problems have grown, distributed and fully-decentralized optimization models have gained attention for providing computational scalability to optimization tools. However, in applications where consumer devices are trusted to serve as distributed computing nodes, compromised devices can expose the optimization algorithm to cybersecurity threats which have not been examined in previous literature. This paper examines potential attack vectors for generalized distributed optimization problems, with a focus on the Alternating…
Recently, distributed algorithms for power system state estimation have attracted significant attention. Along with such advantages as decomposition, parallelization of the original problem and absence of a central computation unit, distributed state estimation may also serve for local information privacy reasons since the only information to be transferred is the boundary states of neighboring areas. In this paper, we propose some novel approaches for speeding up the ADMM-based distributed state estimation algorithms by utilizing some recent results in optimization theory. We also thoroughly …
An alternating direction method of multipliers (ADMM) solver is described for optimal resource allocation problems with separable convex quadratic costs and constraints and linear coupling constraints. We describe a parallel implementation of the solver on a graphics processing unit (GPU) using a bespoke quartic function minimizer. An application to robust optimal energy management in hybrid electric vehicles is described, and the results of numerical simulations comparing the computation times of the parallel GPU implementation with those of an equivalent serial implementation are presented.
The alternating direction method of multipliers (ADMM) is a popular method for solving convex separable minimization problems with linear equality constraints. The generalization of the two-block ADMM to the three-block ADMM is not trivial since the three-block ADMM is not convergence in general. Many variants of three-block ADMM have been developed with guarantee convergence. Besides the ADMM, the alternating minimization algorithm (AMA) is also an important algorithm for solving the convex separable minimization problem with linear equality constraints. The AMA is first proposed by Tseng, an…
Solving optimization problems in multi-agent networks where each agent only has partial knowledge of the problem has become an increasingly important problem. In this paper we consider the problem of minimizing the sum of $n$ convex functions. We assume that each function is only known by one agent. We show that Generalized Distributed ADMM converges Q-linearly to the solution of the mentioned optimization problem if the over all objective function is strongly convex but the functions known by each agent are allowed to be only convex. Establishing Q-linear convergence allows for tracking state…
The alternating direction method of multipliers (ADMM) is a widely used method for solving many convex minimization models arising in signal and image processing. In this paper, we propose an inertial ADMM for solving a two-block separable convex minimization problem with linear equality constraints. This algorithm is obtained by making use of the inertial Douglas-Rachford splitting algorithm to the corresponding dual of the primal problem. We study the convergence analysis of the proposed algorithm in infinite-dimensional Hilbert spaces. Furthermore, we apply the proposed algorithm on the rob…
We propose a convex distributed optimization algorithm for synthesizing robust controllers for large-scale continuous time systems subject to exogenous disturbances. Given a large scale system, instead of solving the larger centralized synthesis task, we decompose the problem into a set of smaller synthesis problems for the local subsystems with a given interconnection topology. Hence, the synthesis problem is constrained to the sparsity pattern dictated by the interconnection topology. To this end, for each subsystem, we solve a local dissipation inequality and then check a small-gain like co…
We propose a distributed design method for decentralized control by exploiting the underlying sparsity properties of the problem. Our method is based on chordal decomposition of sparse block matrices and the alternating direction method of multipliers (ADMM). We first apply a classical parameterization technique to restrict the optimal decentralized control into a convex problem that inherits the sparsity pattern of the original problem. The parameterization relies on a notion of strongly decentralized stabilization, and sufficient conditions are discussed to guarantee this notion. Then, chord…
This paper presents the Distributed Primal Outer Approximation (DiPOA) algorithm for solving Sparse Convex Programming (SCP) problems with separable structures, efficiently, and in a decentralized manner. The DiPOA algorithm development consists of embedding the recently proposed Relaxed Hybrid Alternating Direction Method of Multipliers (RH-ADMM) algorithm into the Outer Approximation (OA) algorithm. We also propose two main improvements to control the quality and the number of cutting planes that approximate nonlinear functions. In particular, the RH-ADMM algorithm acts as a distributed nume…
In this work, we consider the distributed optimization problem in which each node has its own convex cost function and can communicate directly only with its neighbors, as determined by a directed communication topology (directed graph or digraph). First, we reformulate the optimization problem so that Alternating Direction Method of Multipliers (ADMM) can be utilized. Then, we propose an algorithm, herein called Distributed Alternating Direction Method of Multipliers using Finite-Time Exact Ratio Consensus (D-ADMM-FTERC), to solve the multi-node convex optimization problem, in which every nod…
This paper proposes a two-level distributed algorithmic framework for solving the AC optimal power flow (OPF) problem with convergence guarantees. The presence of highly nonconvex constraints in OPF poses significant challenges to distributed algorithms based on the alternating direction method of multipliers (ADMM). In particular, convergence is not provably guaranteed for nonconvex network optimization problems like AC OPF. In order to overcome this difficulty, we propose a new distributed reformulation for AC OPF and a two-level ADMM algorithm that goes beyond the standard framework of ADMM…
We consider a class of distributed optimization problem where the objective function consists of a sum of strongly convex and smooth functions and a (possibly nonsmooth) convex regularizer. A multi-agent network is assumed, where each agent holds a private cost function and cooperates with its neighbors to compute the optimum of the aggregate objective. We propose a quasi-Newton Alternating Direction Method of Multipliers (ADMM) where the primal update is solved inexactly with approximated curvature information. By introducing an intermediate consensus variable, we achieve a block diagonal Hes…
In recent years, several convergent multi-block variants of the alternating direction method of multipliers (ADMM) have been proposed for solving the convex quadratic semidefinite programming via its dual, which is naturally a 3-block separable convex optimization problem with one coupled linear equality constraint. Among of these ADMM-type algorithms, the modified 3-block ADMM in [Chang et al., Neurocomput. 214: 575--586 (2016)] bears a peculiar feature that the augmented Lagrangian function is not necessarily to be minimized with respect to the block-variable corresponding to the quadratic t…
In this work we focus on the problem of minimizing the sum of convex cost functions in a distributed fashion over a peer-to-peer network. In particular, we are interested in the case in which communications between nodes are prone to failures and the agents are not synchronized among themselves. We address the problem proposing a modified version of the relaxed ADMM, which corresponds to the Peaceman-Rachford splitting method applied to the dual. By exploiting results from operator theory, we are able to prove the almost sure convergence of the proposed algorithm under general assumptions on t…
CADMM-Prox: A Bi-level Consensus ADMM for Non-smooth Non-convex Distributed Consensus Optimization
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；North China University of Water Resources and Electric Power（华北水利水电大学）；KTH Royal Institute of Technology（皇家理工学院）
The Alternating Direction Method of Multipliers (ADMM) has gained a lot of attention for solving large-scale and objective-separable constrained optimization. However, the two-block variable structure of the ADMM still limits the practical computational efficiency of the method, because one big matrix factorization is needed at least once even for linear and convex quadratic programming. This drawback may be overcome by enforcing a multi-block structure of the decision variables in the original optimization problem. Unfortunately, the multi-block ADMM, with more than two blocks, is not guarant…
We consider cooperative multi-agent resource sharing problems over time-varying communication networks, where only local communications are allowed. The objective is to minimize the sum of agent-specific composite convex functions subject to a conic constraint that couples agents' decisions. We propose a distributed primal-dual algorithm DPDA-D to solve the saddle point formulation of the sharing problem on time-varying (un)directed communication networks; and we show that primal-dual iterate sequence converges to a point defined by a primal optimal solution and a consensual dual price for the…
This paper considers the problem of distributed model fitting using the alternating directions method of multipliers (ADMM). ADMM splits the learning problem into several smaller subproblems, usually by partitioning the data samples. The different subproblems can be solved in parallel by a set of worker computing nodes coordinated by a master node, and the subproblems are repeatedly solved until convergence. At each iteration, the worker nodes must solve a convex optimization problem whose difficulty increases with the size of the problem. In this paper, we propose a sensitivity-assisted ADMM …
Alternating Direction Method of Multipliers for nonlinear constrained convex problems and applications to distributed resource allocation and constrained machine learning
