The recently developed Distributed Block Proximal Method, for solving stochastic big-data convex optimization problems, is studied in this paper under the assumption of constant stepsizes and strongly convex (possibly non-smooth) local objective functions. This class of problems arises in many learning and classification problems in which, for example, strongly-convex regularizing functions are included in the objective function, the decision variable is extremely high dimensional, and large datasets are employed. The algorithm produces local estimates by means of block-wise updates and communication among the agents. The expected distance from the (global) optimum, in terms of cost value, is shown to decay linearly to a constant value which is proportional to the selected local stepsizes. A numerical example involving a classification problem corroborates the theoretical results.
This paper expands the Cyclic Block Proximal Gradient method for block separable composite minimization by allowing for inexactly computed gradients and proximal maps. The resultant algorithm, the Inexact Cyclic Block Proximal Gradient (I-CBPG) method, shares the same convergence rate as its exactly computed analogue provided the allowable errors decrease sufficiently quickly or are pre-selected to be sufficiently small. We provide numerical experiments that showcase the practical computational advantage of I-CBPG for certain fixed tolerances of approximation error and for a dynamically decrea…
This paper aims to address distributed optimization problems over directed and time-varying networks, where the global objective function consists of a sum of locally accessible convex objective functions subject to a feasible set constraint and coupled inequality constraints whose information is only partially accessible to each agent. For this problem, a distributed proximal-based algorithm, called distributed proximal primal-dual (DPPD) algorithm, is proposed based on the celebrated centralized proximal point algorithm. It is shown that the proposed algorithm can lead to the global optimal …
This note studies the distributed non-convex optimization problem with non-smooth regularization, which has wide applications in decentralized learning, estimation and control. The objective function is the sum of different local objective functions, which consist of differentiable (possibly non-convex) cost functions and non-smooth convex functions. This paper presents a distributed proximal gradient algorithm for the non-smooth non-convex optimization problem over time-varying multi-agent networks. Each agent updates local variable estimate by the multi-step consensus operator and the proxim…
A corrected semi-proximal ADMM for multi-block convex optimization and its application to DNN-SDPs
本文提出了一种修正的半近似乘子交替方向法（alternating direction method of multiplers，ADMM），用于求解一般的线性约束$p$ -块$(p\!\ge 3)$凸优化问题，旨在解决几乎所有现有的直接扩展ADMM的修正版本，虽然有收敛保证，但性能往往比没有收敛保证的直接扩展ADMM本身差得多的困境.具体地说，在每次迭代中，我们使用步长至少为$1$的多块半邻近ADMM作为预测步长，生成一个好的预测点，然后对预测点的中间$(p\!-\!2)$块进行尽可能小的校正。其中，多块半邻近ADMM的步长由当前半邻近ADMM步骤对于上一个校正步骤产生的步骤的不可行性比率自适应地确定。对于所提出的修正半近似ADMM，我们在一个温和的假设下建立了全局收敛性结果，并将其应用于一类重要的具有多个线性等式和/或不等式约束的双非负半定规划（DNN-SDP）问题.我们广泛的数值试验表明，修正的半近端ADMM优于直接扩展ADMM与步长$τ=1.618$和多块ADMM与高斯回代\cite{HTY12,HY13}。它在与直接扩展ADMM相当的计算时间内，对$70\%$测试实例所需迭代次数最少，对约1000个$40\%$测试问题，其迭代次数仅为多块高斯回代ADMM的迭代次数之和$67\%$.
Loopless Proximal Riemannian Gradient EXTRA for Distributed Optimization on Compact Manifolds
School of Intelligent Engineering, Sun Yat-Sen University（中山大学智能工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Department of Mechanical Engineering, University of Victoria（维多利亚大学机械工程系）；The Key Laboratory of Autonomous Intelligent Unmanned Systems, Harbin Institute of Technology（哈尔滨工业大学自主智能无人系统重点实验室）
We design inexact proximal augmented Lagrangian based decomposition methods for convex composite programming problems with dual block-angular structures. Our methods are particularly well suited for convex quadratic programming problems arising from stochastic programming models. The algorithmic framework is based on the application of the abstract inexact proximal ADMM framework developed in [Chen, Sun, Toh, Math. Prog. 161:237--270] to the dual of the target problem, as well as the application of the recently developed symmetric Gauss-Seidel decomposition theorem for solving a proximal multi…
In this paper, we aim to solve a distributed optimization problem with affine coupling constraints in a multi-agent network, where the cost function of the agents is composed of smooth and possibly non-smooth parts. To solve this problem, we resort to the dual problem by deriving the Fenchel conjugate, resulting in a consensus-based constrained optimization problem. Then, we propose a distributed dual proximal gradient algorithm, where the agents make decisions based on the information of immediate neighbors. Provided that the non-smooth parts in the primal cost functions are with some simple …
In this work, we consider solving a distributed optimization problem in a multi-agent network with multiple clusters. In each cluster, the involved agents cooperatively optimize a separable composite function with a common decision variable. Meanwhile, a global cost function of the whole network is considered associated with an affine coupling constraint across the clusters. To solve this problem, we propose a cluster-based dual proximal gradient algorithm by resorting to the dual problem, where the global cost function is optimized when the agents in each cluster achieve an agreement on the o…
We show that the asynchronous block-iterative primal-dual projective splitting framework introduced by P. L. Combettes and J. Eckstein in their 2018 Math. Program. paper can be viewed as an instantiation of the recently proposed warped proximal algorithm.
A Stochastic Block-coordinate Proximal Newton Method for Nonconvex Composite Minimization
School of Mathematical Sciences, Jiangsu University（江苏大学数学科学学院）；Center of Excellence for Generative AI, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学生成式人工智能卓越中心）
In this paper we introduce a class of novel distributed algorithms for solving stochastic big-data convex optimization problems over directed graphs. In the addressed set-up, the dimension of the decision variable can be extremely high and the objective function can be nonsmooth. The general algorithm consists of two main steps: a consensus step and an update on a single block of the optimization variable, which is then broadcast to neighbors. Three special instances of the proposed method, involving particular problem structures, are then presented. In the general case, the convergence of a d…
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
Affine-coupled Distributed Optimization via Distributed Proximal Jacobian ADMM with Quantized Communication
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Bologna（博洛尼亚大学）；KTH Royal Institute of Technology（皇家理工学院）；The Hong Kong University of Science and Technology（香港科技大学）
We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function, each of the nonsmooth summands depending on an independent block variable, and a smooth function which couples the two block variables. The algorithm is a full splitting method, which means that the nonsmooth functions are processed via their proximal operators, the smooth function via gradient steps, and the linear operator via matrix times vector multiplication. We provide sufficient conditions for the bou…
A Unifying Primal-Dual Proximal Framework for Distributed Nonconvex Optimization
School of Information Science and Technology, Shanghaitech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Energy Efficient and Custon AI IC（上海市能效与定制人工智能集成电路工程研究中心）
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization. We derive sublinear and linear convergence results with new rates on the function value suboptimality or distance to the solution, as well as new accelerated versions, using varying stepsizes. In addition, we propose distributed variants of these algorithms, which can be accelerated as well. While most existing results are ergodic, our nonergodic results significantly broaden our understanding of primal-dual optimization algorithms.
