In this paper, we propose a second-order continuous primal-dual dynamical system with time-dependent positive damping terms for a separable convex optimization problem with linear equality constraints. By the Lyapunov function approach, we investigate asymptotic properties of the proposed dynamical system as the time $t\to+\infty$. The convergence rates are derived for different choices of the damping coefficients. We also show that the obtained results are robust under external perturbations.
In this paper, we propose an inertial accelerated primal-dual method for the linear equality constrained convex optimization problem. When the objective function has a ``nonsmooth + smooth'' composite structure, we further propose an inexact inertial primal-dual method by linearizing the smooth individual function and solving the subproblem inexactly. Assuming merely convexity, we prove that the proposed methods enjoy $\mathcal{O}(1/k^2)$ convergence rate on the objective residual and the feasibility violation in the primal model. Numerical results are reported to demonstrate the validity of t…
We consider an inertial primal-dual algorithm to compute the minimizations of the sum of two convex functions and the composition of another convex function with a continuous linear operator. With the idea of coordinate descent, we design a stochastic coordinate descent inertial primal-dual splitting algorithm. Moreover, in order to prove the convergence of the proposed inertial algorithm, we formulate first the inertial version of the randomized Krasnosel'skii-Mann iterations algorithm for approximating the set of fixed points of a nonexpansive operator and investigate its convergence propert…
We propose and study a novel stochastic inertial primal-dual approach to solve composite optimization problems. These latter problems arise naturally when learning with penalized regularization schemes. Our analysis provide convergence results in a general setting, that allows to analyze in a unified framework a variety of special cases of interest. Key in our analysis is considering the framework of splitting algorithm for solving a monotone inclusions in suitable product spaces and for a specific choice of preconditioning operators.
We study inertial versions of primal-dual proximal splitting, also known as the Chambolle--Pock method. Our starting point is the preconditioned proximal point formulation of this method. By adding correctors corresponding to the anti-symmetric part of the relevant monotone operator, using a FISTA-style gap unrolling argument, we are able to derive gap estimates instead of merely ergodic gap estimates. Moreover, based on adding a diagonal component to this corrector, we are able to combine strong convexity based acceleration with inertial acceleration. We test our proposed method on image proc…
We consider an inertial primal-dual fixed point algorithm (IPDFP) to compute the minimizations of the following Problem (1.1). This is a full splitting approach, in the sense that the nonsmooth functions are processed individually via their proximity operators. The convergence of the IPDFP is obtained by reformulating the Problem (1.1) to the sum of three convex functions. This work brings together and notably extends several classical splitting schemes, like the primaldual method proposed by Chambolle and Pock, and the recent proximity algorithms of Charles A. et al designed for the L1/TV ima…
Fast convergence rates and trajectory convergence of a Tikhonov regularized inertial primal\mbox{-}dual dynamical system with time scaling and vanishing damping
The primal-dual algorithm recently proposed by Chambolle & Pock (abbreviated as CPA) for structured convex optimization is very efficient and popular. It was shown by Chambolle & Pock in \cite{CP11} and also by Shefi & Teboulle in \cite{ST14} that CPA and variants are closely related to preconditioned versions of the popular alternating direction method of multipliers (abbreviated as ADM). In this paper, we further clarify this connection and show that CPAs generate exactly the same sequence of points with the so-called linearized ADM (abbreviated as LADM) applied to either the primal problem …
Many problems arising in image processing and signal recovery with multi-regularization can be formulated as minimization of a sum of three convex separable functions. Typically, the objective function involves a smooth function with Lipschitz continuous gradient, a linear composite nonsmooth function and a nonsmooth function. In this paper, we propose a primal-dual fixed-point (PDFP) scheme to solve the above class of problems. The proposed algorithm for three block problems is a fully splitting symmetric scheme, only involving explicit gradient and linear operators without inner iteration, w…
This paper presents the Distributed Primal Outer Approximation (DiPOA) algorithm for solving Sparse Convex Programming (SCP) problems with separable structures, efficiently, and in a decentralized manner. The DiPOA algorithm development consists of embedding the recently proposed Relaxed Hybrid Alternating Direction Method of Multipliers (RH-ADMM) algorithm into the Outer Approximation (OA) algorithm. We also propose two main improvements to control the quality and the number of cutting planes that approximate nonlinear functions. In particular, the RH-ADMM algorithm acts as a distributed nume…
In this paper we combine the stochastic variance reduced gradient (SVRG) method [17] with the primal dual fixed point method (PDFP) proposed in [7] to solve a sum of two convex functions and one of which is linearly composite. This type of problems are typically arisen in sparse signal and image reconstruction. The proposed SVRG-PDFP can be seen as a generalization of Prox-SVRG [37] originally designed for the minimization of a sum of two convex functions. Based on some standard assumptions, we propose two variants, one is for strongly convex objective function and the other is for general con…
This paper introduces a coordinate descent version of the Vũ-Condat algorithm. By coordinate descent, we mean that only a subset of the coordinates of the primal and dual iterates is updated at each iteration, the other coordinates being maintained to their past value. Our method allows us to solve optimization problems with a combination of differentiable functions, constraints as well as non-separable and non-differentiable regularizers. We show that the sequences generated by our algorithm converge to a saddle point of the problem at stake, for a wider range of parameter values than previou…
We introduce and investigate the convergence properties of an inertial forward-backward-forward splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitzian operator. By making use of the product space approach, we expand it to the solving of inclusion problems involving mixtures of linearly composed and parallel-sum type monotone operators. We obtain in this way an inertial forward-backward-forward primal-dual splitting algorithm having as main characteristic the fact that in the iterative scheme all operators a…
We propose a novel dynamically weighted inertial forward-backward algorithm (DWIFOB) for solving structured monotone inclusion problems. The scheme exploits the globally convergent forward-backward algorithm with deviations in [26] as the basis and combines it with the extrapolation technique used in Anderson acceleration to improve local convergence. We also present a globally convergent primal-dual variant of DWIFOB and numerically compare its performance to the primal-dual method of Chambolle-Pock and a Tikhonov regularized version of Anderson acceleration applied to the same problem. In al…
The alternating direction method of multipliers (ADMM) is a widely used method for solving many convex minimization models arising in signal and image processing. In this paper, we propose an inertial ADMM for solving a two-block separable convex minimization problem with linear equality constraints. This algorithm is obtained by making use of the inertial Douglas-Rachford splitting algorithm to the corresponding dual of the primal problem. We study the convergence analysis of the proposed algorithm in infinite-dimensional Hilbert spaces. Furthermore, we apply the proposed algorithm on the rob…
This paper considers large scale constrained convex (possibly composite and non-separable) programs, which are usually difficult to solve by interior point methods or other Newton-type methods due to the non-smoothness or the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, they are often solved by first order gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, also known as the Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with an $O(1/ε^2)$ convergence time. Recently, a new …
In this paper we propose a distributed dual gradient algorithm for minimizing linearly constrained separable convex problems and analyze its rate of convergence. In particular, we prove that under the assumption of strong convexity and Lipshitz continuity of the gradient of the primal objective function we have a global error bound type property for the dual problem. Using this error bound property we devise a fully distributed dual gradient scheme, i.e. a gradient scheme based on a weighted step size, for which we derive global linear rate of convergence for both dual and primal suboptimality…
In Hilbert space, we propose a family of primal-dual dynamical system for affine constrained convex optimization problem. Several damping coefficients, time scaling coefficients, and perturbation terms are thus considered. By constructing the energy functions, we investigate the convergence rates with different choices of the damping coefficients and time scaling coefficients. Our results extend the inertial dynamical approaches for unconstrained convex optimization problems to affine constrained convex optimization problems.
In this paper, we develop a chance-constrained decentralized electric vehicle (EV) charging control framework to achieve "valley-filling" meanwhile meeting individual charging requirements and satisfying distribution network constraints. The control design is formulated as an optimization problem with a stochastic non-separable objective function and globally coupled chance constraints. We propose a novel chance-constrained shrunken-primal-dual subgradient (CC-SPDS) algorithm to support the chance-constrained decentralized control scheme and verify its efficacy and convergence with a represent…
By time discretization of a second-order primal-dual dynamical system with damping $α/t$ where an inertial construction in the sense of Nesterov is needed only for the primal variable, we propose a fast primal-dual algorithm for a linear equality constrained convex optimization problem. Under a suitable scaling condition, we show that the proposed algorithm enjoys a fast convergence rate for the objective residual and the feasibility violation, and the decay rate can reach $\mathcal{O}(1/k^{α-1})$ at the most. We also study convergence properties of the corresponding primal-dual dynamical syst…
We consider the problem of minimizing a convex, separable, nonsmooth function subject to linear constraints. The numerical method we propose is a block-coordinate extension of the Chambolle-Pock primal-dual algorithm. We prove convergence of the method without resorting to assumptions like smoothness or strong convexity of the objective, full-rank condition on the matrix, strong duality or even consistency of the linear system. Freedom from imposing the latter assumption permits convergence guarantees for misspecified or noisy systems.
We propose an inertial forward-backward splitting algorithm to compute the zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal-dual splitting methods for solving structured systems of composite monotone inclusions and prove their convergence. Our results extend to the stochastic and inertial setting various types of structure…
Second-order dynamical systems are important tools for solving optimization problems, and most of existing works in this field have focused on unconstrained optimization problems. In this paper, we propose an inertial primal-dual dynamical system with constant viscous damping and time scaling for the linear equality constrained convex optimization problem, which consists of a second-order ODE for the primal variable and a first-order ODE for the dual variable. When the scaling satisfies certain conditions, we prove its convergence property without assuming strong convexity. Even the convergenc…
In this paper we propose distributed dual gradient algorithms for linearly constrained separable convex problems and analyze their rate of convergence under different assumptions. Under the strong convexity assumption on the primal objective function we propose two distributed dual fast gradient schemes for which we prove sublinear rate of convergence for dual suboptimality but also primal suboptimality and feasibility violation for an average primal sequence or for the last generated primal iterate. Under the additional assumption of Lipshitz continuity of the gradient of the primal objective…
In this paper, we propose a randomized intertial block-coordinate primaldual fixed point algorithm to solve a wide array of monotone inclusion problems base on the modification of the heavy ball method of Nesterov. These methods rely on a sweep of blocks of variables which are activated at each iteration according to a random rule. To this end we formulate the inertial version of the Krasnosel'skii-Mann algorithm for approximating the set of fixed points of a quasinonexpansive operator, for which we also provide an exhaustive convergence analysis. As a by-product, we can obtain some intertial …
We study nonconvex quadratic problems (QPs) with quadratic separable constraints, where these constraints can be defined both as inequalities or equalities. We derive sufficient conditions for these types of problems to present the S-property, which ultimately guarantees strong duality between the primal and dual problems of the QP. We study the existence of solutions and propose a novel distributed algorithm to solve the problem optimally when the S-property is satisfied. Finally, we illustrate our theoretical results proving that the robust least squares problem with multiple constraints has…
In this paper, we first propose a general inertial proximal point method for the mixed variational inequality (VI) problem. Based on our knowledge, without stronger assumptions, convergence rate result is not known in the literature for inertial type proximal point methods. Under certain conditions, we are able to establish the global convergence and a $o(1/k)$ convergence rate result (under certain measure) of the proposed general inertial proximal point method. We then show that the linearized alternating direction method of multipliers (ADMM) for separable convex optimization with linear co…
We propose a new modified primal-dual proximal best approximation method for solving convex not necessarily differentiable optimization problems. The novelty of the method relies on introducing memory by taking into account iterates computed in previous steps in the formulas defining current iterate. To this end we consider projections onto intersections of halfspaces generated on the basis of the current as well as the previous iterates. To calculate these projections we are using recently obtained closed-form expressions for projectors onto polyhedral sets. The resulting algorithm with memor…
Dual decomposition has been successfully employed in a variety of distributed convex optimization problems solved by a network of computing and communicating nodes. Often, when the cost function is separable but the constraints are coupled, the dual decomposition scheme involves local parallel subgradient calculations and a global subgradient update performed by a master node. In this paper, we propose a consensus-based dual decomposition to remove the need for such a master node and still enable the computing nodes to generate an approximate dual solution for the underlying convex optimizatio…
This work aims to minimize a continuously differentiable convex function with Lipschitz continuous gradient under linear equality constraints. The proposed inertial algorithm results from the discretization of the second-order primal-dual dynamical system with asymptotically vanishing damping term addressed by Bot and Nguyen in [Bot, Nguyen, JDE, 2021], and it is formulated in terms of the Augmented Lagrangian associated with the minimization problem. The general setting we consider for the inertial parameters covers the three classical rules by Nesterov, Chambolle-Dossal and Attouch-Cabot use…
In this paper we propose a new inexact dual decomposition algorithm for solving separable convex optimization problems. This algorithm is a combination of three techniques: dual Lagrangian decomposition, smoothing and excessive gap. The algorithm requires only one primal step and two dual steps at each iteration and allows one to solve the subproblem of each component inexactly and in parallel. Moreover, the algorithmic parameters are updated automatically without any tuning strategy as in augmented Lagrangian approaches. We analyze the convergence of the algorithm and estimate its $O(\frac{1}…
In this paper, a centralized two-block separable optimization is considered for which a fully parallel primal-dual discrete-time algorithm with fixed step size is derived based on monotone operator splitting method. In this algorithm, the primal variables are updated in an alternating fashion like Alternating Direction Method of Multipliers (ADMM). However, unlike existing discrete-time algorithms such as Method of Multipliers (MM), ADMM, Bi-Alternating Direction Method of Multipliers (BiADMM), and Primal-Dual Fixed Point (PDFP) algorithms, that all suffer from sequential updates, all primal a…
We design accelerated algorithms with improved rates for several fundamental classes of optimization problems. Our algorithms all build upon techniques related to the analysis of primal-dual extragradient methods via relative Lipschitzness proposed recently by [CST21]. (1) Separable minimax optimization. We study separable minimax optimization problems $\min_x \max_y f(x) - g(y) + h(x, y)$, where $f$ and $g$ have smoothness and strong convexity parameters $(L^x, μ^x)$, $(L^y, μ^y)$, and $h$ is convex-concave with a $(Λ^{xx}, Λ^{xy}, Λ^{yy})$-blockwise operator norm bounded Hessian. We provide …
We extend a primal-dual fixed point algorithm (PDFP) proposed in [5] to solve two kinds of separable multi-block minimization problems, arising in signal processing and imaging science. This work shows the flexibility of applying PDFP algorithm to multi-block problems and illustrate how practical and fully decoupled schemes can be derived, especially for parallel implementation of large scale problems. The connections and comparisons to the alternating direction method of multiplier (ADMM) are also present. We demonstrate how different algorithms can be obtained by splitting the problems in di…
We investigate the convergence properties of a stochastic primal-dual splitting algorithm for solving structured monotone inclusions involving the sum of a cocoercive operator and a composite monotone operator. The proposed method is the stochastic extension to monotone inclusions of a proximal method studied in {\em Y. Drori, S. Sabach, and M. Teboulle, A simple algorithm for a class of nonsmooth convex-concave saddle-point problems, 2015} and {\em I. Loris and C. Verhoeven, On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty, 2011} for saddl…
The alternating direction method of multipliers (ADMM) is a popular method for solving convex separable minimization problems with linear equality constraints. The generalization of the two-block ADMM to the three-block ADMM is not trivial since the three-block ADMM is not convergence in general. Many variants of three-block ADMM have been developed with guarantee convergence. Besides the ADMM, the alternating minimization algorithm (AMA) is also an important algorithm for solving the convex separable minimization problem with linear equality constraints. The AMA is first proposed by Tseng, an…
We study deterministic and stochastic primal-dual sub-gradient algorithms for distributed optimization of a separable objective function with global inequality constraints. In both algorithms, the norm of the Lagrangian multipliers are controlled by augmenting the corresponding Lagrangian function with a quadratic regularization term. Specifically, we show that when the stepsize of each algorithm satisfies a certain restriction, the norm of the Lagrangian multipliers is upper bounded by an expression that is inversely proportional to the parameter of the regularization. We use this result to c…
Multi-block separable convex problems recently received considerable attention. This class of optimization problems minimizes a separable convex objective function with linear constraints. The algorithmic challenges come from the fact that the classic alternating direction method of multipliers (ADMM) for the problem is not necessarily convergent. However, it is observed that ADMM outperforms numerically many of its variants with guaranteed theoretical convergence. The goal of this paper is to develop convergent and computationally efficient algorithms for solving multi-block separable convex …
This paper considers large scale constrained convex programs, which are usually not solvable by interior point methods or other Newton-type methods due to the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, large scale constrained convex programs are often solved by gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, aka, Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with the $O(1/\sqrt{t})$ convergence rate, where $t$ is the number of iterations. If the objective and constra…
