Heavy computational load for solving nonconvex problems for large-scale systems or systems with real-time demands at each sample step has been recognized as one of the reasons for preventing a wider application of nonlinear model predictive control (NMPC). To improve the real-time feasibility of NMPC with input nonlinearity, we devise an innovative scheme called cloud-based computational model predictive control (MPC) by using an elaborately designed parallel multi-block alternating direction method of multipliers (ADMM) algorithm. This novel parallel multi-block ADMM algorithm is tailored to tackle the computational issue of solving a nonconvex problem with nonlinear constraints.
The paper presents a distributed model predictive control (DMPC) scheme for continuous-time nonlinear systems based on the alternating direction method of multipliers (ADMM). A stopping criterion in the ADMM algorithm limits the iterations and therefore the required communication effort during the distributed MPC solution at the expense of a suboptimal solution. Stability results are presented for the suboptimal DMPC scheme under two different ADMM convergence assumptions. In particular, it is shown that the required iterations in each ADMM step are bounded, which is also confirmed in simulati…
This article presents a sparse, low-memory footprint optimization algorithm for the implementation of the model predictive control (MPC) for tracking formulation in embedded systems. This MPC formulation has several advantages over standard MPC formulations, such as an increased domain of attraction and guaranteed recursive feasibility even in the event of a sudden reference change. However, this comes at the expense of the addition of a small amount of decision variables to the MPC's optimization problem that complicates the structure of its matrices. We propose a sparse optimization algorith…
The multi-block ADMM has received much attention from optimization researchers due to its excellent scalability. In this paper, the multi-block ADMM is applied to solve two large-scale problems related to isotonic regression. Numerical experiments show that the multi-block ADMM is convergent when the chosen parameter is small enough and the multi-block ADMM scales well compared with baselines.
In recent years, several convergent multi-block variants of the alternating direction method of multipliers (ADMM) have been proposed for solving the convex quadratic semidefinite programming via its dual, which is naturally a 3-block separable convex optimization problem with one coupled linear equality constraint. Among of these ADMM-type algorithms, the modified 3-block ADMM in [Chang et al., Neurocomput. 214: 575--586 (2016)] bears a peculiar feature that the augmented Lagrangian function is not necessarily to be minimized with respect to the block-variable corresponding to the quadratic t…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；College of Humanities and Sciences（人文与科学学院）；Northeast Normal University（东北师范大学）；University of Minnesota, Twin Cities（明尼苏达大学双城分校）
Alternating Direction Method of Multipliers (ADMM) has become a widely used optimization method for convex problems, particularly in the context of data mining in which large optimization problems are often encountered. ADMM has several desirable properties, including the ability to decompose large problems into smaller tractable sub-problems and ease of parallelization, that are essential in these scenarios. The most common form of ADMM is the two-block, in which two sets of primal variables are updated alternatingly. Recent years have seen advances in multi-block ADMM, which update more than…
This paper establishes the iteration-complexity of a Jacobi-type non-Euclidean proximal alternating direction method of multipliers (ADMM) for solving multi-block linearly constrained nonconvex programs. The subproblems of this ADMM variant can be solved in parallel and hence the method has great potential to solve large scale multi-block linearly constrained nonconvex programs. Moreover, our analysis allows the Lagrange multiplier to be updated with a relaxation parameter in the interval (0, 2).
A corrected semi-proximal ADMM for multi-block convex optimization and its application to DNN-SDPs
本文提出了一种修正的半近似乘子交替方向法（alternating direction method of multiplers，ADMM），用于求解一般的线性约束$p$ -块$(p\!\ge 3)$凸优化问题，旨在解决几乎所有现有的直接扩展ADMM的修正版本，虽然有收敛保证，但性能往往比没有收敛保证的直接扩展ADMM本身差得多的困境.具体地说，在每次迭代中，我们使用步长至少为$1$的多块半邻近ADMM作为预测步长，生成一个好的预测点，然后对预测点的中间$(p\!-\!2)$块进行尽可能小的校正。其中，多块半邻近ADMM的步长由当前半邻近ADMM步骤对于上一个校正步骤产生的步骤的不可行性比率自适应地确定。对于所提出的修正半近似ADMM，我们在一个温和的假设下建立了全局收敛性结果，并将其应用于一类重要的具有多个线性等式和/或不等式约束的双非负半定规划（DNN-SDP）问题.我们广泛的数值试验表明，修正的半近端ADMM优于直接扩展ADMM与步长$τ=1.618$和多块ADMM与高斯回代\cite{HTY12,HY13}。它在与直接扩展ADMM相当的计算时间内，对$70\%$测试实例所需迭代次数最少，对约1000个$40\%$测试问题，其迭代次数仅为多块高斯回代ADMM的迭代次数之和$67\%$.
Embedding randomization procedures in the Alternating Direction Method of Multipliers (ADMM) has recently attracted an increasing amount of interest as a remedy to the fact that the direct multi-block generalization of ADMM is not necessarily convergent. Even if, in practice, the introduction of such techniques could \textit{mitigate} the diverging behaviour of the multi-block extension of ADMM, from the theoretical point of view, it can ensure just the \textit{convergence in expectation}, which may not be a good indicator of its robustness and efficiency. In this work, analysing the strongly …
UC Berkeley（加州大学伯克利分校）；UC Davis（加州大学戴维斯分校）；University of Minnesota（明尼苏达大学）
The alternating direction method of multipliers (ADMM) has been successfully applied to solve structured convex optimization problems due to its superior practical performance. The convergence properties of the 2-block ADMM have been studied extensively in the literature. Specifically, it has been proven that the 2-block ADMM globally converges for any penalty parameter $γ>0$. In this sense, the 2-block ADMM allows the parameter to be free, i.e., there is no need to restrict the value for the parameter when implementing this algorithm in order to ensure convergence. However, for the 3-block AD…
For multi-block alternating direction method of multipliers(ADMM), where the objective function can be decomposed into multiple block components, we show that with block symmetric Gauss-Seidel iteration, the algorithm will converge quickly. The method will apply a block symmetric Gauss-Seidel iteration in the primal update and a linear correction that can be derived in view of Richard iteration. We also establish the linear convergence rate for linear systems.
In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proxima…
This paper aims to present a fairly accessible generalization of several symmetric Gauss-Seidel decomposition based multi-block proximal alternating direction methods of multipliers (ADMMs) for convex composite optimization problems. The proposed method unifies and refines many constructive techniques that were separately developed for the computational efficiency of multi-block ADMM-type algorithms. Specifically, the majorized augmented Lagrangian functions, the indefinite proximal terms, the inexact symmetric Gauss-Seidel decomposition theorem, the tolerance criteria of approximately solving…
We study the solution of block-structured linear algebra systems arising in optimization by using iterative solution techniques. These systems are the core computational bottleneck of many problems of interest such as parameter estimation, optimal control, network optimization, and stochastic programming. Our approach uses a Krylov solver (GMRES) that is preconditioned with an alternating method of multipliers (ADMM). We show that this ADMM-GMRES approach overcomes well-known scalability issues of Schur complement decomposition in problems that exhibit a high degree of coupling. The effectiven…
We consider demand-side primary frequency control in the power grid provided by smart and flexible loads: loads change consumption to match generation and help the grid while minimizing disutility for consumers incurred by consumption changes. The dual formulation of this problem has been solved previously by Zhao et al. in a decentralized manner for consumer disutilities that are twice continuously differentiable with respect to consumption changes. In this work, we propose a decentralized multi-block alternating-direction-method-of-multipliers (DM-ADMM) algorithm to solve this problem. In co…
The Alternating Direction Method of Multipliers (ADMM) has gained a lot of attention for solving large-scale and objective-separable constrained optimization. However, the two-block variable structure of the ADMM still limits the practical computational efficiency of the method, because one big matrix factorization is needed at least once even for linear and convex quadratic programming. This drawback may be overcome by enforcing a multi-block structure of the decision variables in the original optimization problem. Unfortunately, the multi-block ADMM, with more than two blocks, is not guarant…
The alternating direction method of multipliers (ADMM) is a popular method for solving convex separable minimization problems with linear equality constraints. The generalization of the two-block ADMM to the three-block ADMM is not trivial since the three-block ADMM is not convergence in general. Many variants of three-block ADMM have been developed with guarantee convergence. Besides the ADMM, the alternating minimization algorithm (AMA) is also an important algorithm for solving the convex separable minimization problem with linear equality constraints. The AMA is first proposed by Tseng, an…
National University of Singapore（新加坡国立大学）；University of California, Berkeley（加州大学伯克利分校）；Temasek Laboratories, National University of Singapore（新加坡国立大学淡马锡实验室）；University of British Columbia（不列颠哥伦比亚大学）
Distributed optimization is often widely attempted and innovated as an attractive and preferred methodology to solve large-scale problems effectively in a localized and coordinated manner. Thus, it is noteworthy that the methodology of distributed model predictive control (DMPC) has become a promising approach to achieve effective outcomes, e.g., in decision-making tasks for multi-agent systems. However, the typical deployment of such distributed MPC frameworks would lead to the involvement of nonlinear processes with a large number of nonconvex constraints. To address this important problem, …
The alternating direction method of multipliers (ADMM) proposed by Glowinski and Marrocco is a benchmark algorithm for two-block separable convex optimization problems with linear equality constraints. It has been modified, specified, and generalized from various perspectives to tackle more concrete or complicated application problems. Despite its versatility and phenomenal popularity, it remains unknown whether or not the ADMM can be extended to separable convex optimization problems with linear inequality constraints. In this paper, we lay down the foundation of how to extend the ADMM to two…
AI-Assisted Discovery and Construction of a Counterexample to the Convergence of Three-Block ADMM with the Identity Matrix as its Third Constraint Block
School of Mathematical Sciences, East China Normal University（华东师范大学数学科学学院）
Full-waveform inversion (FWI) is a high-resolution and computationally intensive imaging technique to reconstruct unknown parameters in the computational model in which the waves propagate; however, an accurate model of only part of this medium is required for some applications. To decrease the computational burden of such problems, target-oriented FWI was proposed where the redatumed data on the part of the medium or localized solvers for the wave equation are used. On the other hand, the classical formulation of FWI suffers from non-linearity and ill-posedness, which makes FWI sensitive to t…
Stanford University（斯坦福大学）；The University of Edinburgh（爱丁堡大学）；Oracle Labs（甲骨文实验室）
The Alternating Direction Method of Multipliers (ADMM) has now days gained tremendous attentions for solving large-scale machine learning and signal processing problems due to the relative simplicity. However, the two-block structure of the classical ADMM still limits the size of the real problems being solved. When one forces a more-than-two-block structure by variable-splitting, the convergence speed slows down greatly as observed in practice. Recently, a randomly assembled cyclic multi-block ADMM (RAC-MBADMM) was developed by the authors for solving general convex and nonconvex quadratic op…
Recently, the alternating direction method of multipliers (ADMM) has found many efficient applications in various areas; and it has been shown that the convergence is not guaranteed when it is directly extended to the multiple-block case of separable convex minimization problems where there are $m\ge 3$ functions without coupled variables in the objective. This fact has given great impetus to investigate various conditions on both the model and the algorithm's parameter that can ensure the convergence of the direct extension of ADMM (abbreviated as "e-ADMM"). Despite some results under very st…
In this paper, we propose an inexact multi-block ADMM-type first-order method for solving a class of high-dimensional convex composite conic optimization problems to moderate accuracy. The design of this method combines an inexact 2-block majorized semi-proximal ADMM and the recent advances in the inexact symmetric Gauss-Seidel (sGS) technique for solving a multi-block convex composite quadratic programming whose objective contains a nonsmooth term involving only the first block-variable. One distinctive feature of our proposed method (the sGS-imsPADMM) is that it only needs one cycle of an in…
The Alternating Direction Method of Multipliers (ADMM) has been proved to be effective for solving separable convex optimization subject to linear constraints. In this paper, we propose a Generalized Symmetric ADMM (GS-ADMM), which updates the Lagrange multiplier twice with suitable stepsizes, to solve the multi-block separable convex programming. This GS-ADMM partitions the data into two group variables so that one group consists of $p$ block variables while the other has $q$ block variables, where $p \ge 1$ and $q \ge 1$ are two integers. The two grouped variables are updated in a {\it Gauss…
Linearized alternating direction method of multipliers (ADMM) as an extension of ADMM has been widely used to solve linearly constrained problems in signal processing, machine leaning, communications, and many other fields. Despite its broad applications in nonconvex optimization, for a great number of nonconvex and nonsmooth objective functions, its theoretical convergence guarantee is still an open problem. In this paper, we propose a two-block linearized ADMM and a multi-block parallel linearized ADMM for problems with nonconvex and nonsmooth objectives. Mathematically, we present that the …
The Potts model has many applications. It is equivalent to some min-cut and max-flow models. Primal-dual algorithms have been used to solve these problems. Due to the special structure of the models, convergence proof is still a difficult problem. In this work, we developed two novel, preconditioned, and over-relaxed alternating direction methods of multipliers (ADMM) with convergence guarantee for these models. Using the proposed preconditioners or block preconditioners, we get accelerations with the over-relaxation variants of preconditioned ADMM. The preconditioned and over-relaxed Douglas-…
In this paper we present a convex formulation of the Model Predictive Control (MPC) optimisation for energy management in hybrid electric vehicles, and an Alternating Direction Method of Multipliers (ADMM) algorithm for its solution. We develop a new proof of convexity for the problem that allows the nonlinear dynamics to be modelled as a linear system, then demonstrate the performance of ADMM in comparison with Dynamic Programming (DP) through simulation. The results demonstrate up to two orders of magnitude improvement in solution time for comparable accuracy against DP.
Solving combinatorial optimization problems on current noisy quantum devices is currently being advocated for (and restricted to) binary polynomial optimization with equality constraints via quantum heuristic approaches. This is achieved using, e.g., the variational quantum eigensolver (VQE) and the quantum approximate optimization algorithm (QAOA). We present a decomposition-based approach to extend the applicability of current approaches to "quadratic plus convex" mixed binary optimization (MBO) problems, so as to solve a broad class of real-world optimization problems. In the MBO framework,…
This paper is concerned with the computing efficiency of model predictive control (MPC) problems for dynamical systems with both rate and amplitude constraints on the inputs. Instead of augmenting the decision variables of the underlying finite-horizon optimal control problem to accommodate the input rate constraints, we propose to solve this problem using the fast gradient method (FGM), where the projection step is solved using Dykstra's algorithm. We show that, relative to the Alternating Direction of Method Multipliers (ADMM), this approach greatly reduces the computation time while halving…
This paper details an investigation into the computational performance of algorithms used for solving a convex formulation of the optimization problem associated with model predictive control for energy management in hybrid electric vehicles with nonlinear losses. A projected interior point method is proposed, where the size and complexity of the Newton step matrix inversion is reduced by applying inequality constraints on the control input as a projection, and its properties are demonstrated through simulation in comparison with an alternating direction method of multipliers (ADMM) algorithm,…
In this paper, we show that for a class of linearly constrained convex composite optimization problems, an (inexact) symmetric Gauss-Seidel based majorized multi-block proximal alternating direction method of multipliers (ADMM) is equivalent to an {\em inexact} proximal augmented Lagrangian method (ALM). This equivalence not only provides new perspectives for understanding some ADMM-type algorithms but also supplies meaningful guidelines on implementing them to achieve better computational efficiency. Even for the two-block case, a by-product of this equivalence is the convergence of the whole…
In this paper we propose an Alternating Direction Method of Multipliers (ADMM) algorithm for solving a Model Predictive Control (MPC) optimization problem, in which the system has state and input constraints and a nonlinear input map. The resulting optimization is nonconvex, and we provide a proof of convergence to a point satisfying necessary conditions for optimality. This general method is proposed as a solution for blended mode control of hybrid electric vehicles, to allow optimization in real time. To demonstrate the properties of the algorithm we conduct numerical experiments on randomly…
