Many large-scale and distributed optimization problems can be brought into a composite form in which the objective function is given by the sum of a smooth term and a nonsmooth regularizer. Such problems can be solved via a proximal gradient method and its variants, thereby generalizing gradient descent to a nonsmooth setup. In this paper, we view proximal algorithms as dynamical systems and leverage techniques from control theory to study their global properties. In particular, for problems with strongly convex objective functions, we utilize the theory of integral quadratic constraints to prove the global exponential stability of the equilibrium points of the differential equations that govern the evolution of proximal gradient and Douglas-Rachford splitting flows. In our analysis, we use the fact that these algorithms can be interpreted as variable-metric gradient methods on the suitable envelopes and exploit structural properties of the nonlinear terms that arise from the gradient of the smooth part of the objective function and the proximal operator associated with the nonsmooth regularizer. We also demonstrate that these envelopes can be obtained from the augmented Lagrangian associated with the original nonsmooth problem and establish conditions for global exponential convergence even in the absence of strong convexity.
The University of Chicago（芝加哥大学）；Peking University（北京大学）
Along with developing of Peaceman-Rachford Splittling Method (PRSM), many batch algorithms based on it have been studied very deeply. But almost no algorithm focused on the performance of stochastic version of PRSM. In this paper, we propose a new stochastic algorithm based on PRSM, prove its convergence rate in ergodic sense, and test its performance on both artificial and real data. We show that our proposed algorithm, Stochastic Scalable PRSM (SS-PRSM), enjoys the $O(1/K)$ convergence rate, which is the same as those newest stochastic algorithms that based on ADMM but faster than general St…
The Douglas-Rachford (DR) algorithm is an iterative procedure that uses sequential reflections onto convex sets and which has become popular for convex feasibility problems. In this paper we propose a structural generalization that allows to use $r$-sets-DR operators in a cyclic fashion. We prove convergence and present numerical illustrations of the potential advantage of such operators with $r>2$ over the classical $2$-sets-DR operators in a cyclic algorithm.
In this paper, we study a parameterized Douglas-Rachford splitting method for a class of nonconvex optimization problem. A new merit function is constructed to establish the convergence of the whole sequence generated by the parameterized Douglas-Rachford splitting method. We then apply the parameterized Douglas-Rachford splitting method to three important classes of nonconvex optimization problems arising in data science: sparsity constrained least squares problem, feasibility problem and low rank matrix completion. Numerical results validate the effectiveness of the parameterized Douglas-Rac…
We introduce and study a geometric modification of the Douglas-Rach\-ford method called the Circumcentered-Douglas-Rachford method. This method iterates by taking the intersection of bisectors of reflection steps for solving certain classes of feasibility problems. The convergence analysis is established for best approximation problems involving two (affine) subspaces and both our theoretical and numerical results compare favorably to the original Douglas-Rachford method. Under suitable conditions, it is shown that the linear rate of convergence of the Circumcentered-Douglas-Rachford method is…
We propose a new approach for analyzing convergence of the Douglas-Rachford splitting method for solving convex composite optimization problems. The approach is based on a continuously differentiable function, the Douglas-Rachford Envelope (DRE), whose stationary points correspond to the solutions of the original (possibly nonsmooth) problem. By proving the equivalence between the Douglas-Rachford splitting method and a scaled gradient method applied to the DRE, results from smooth unconstrained optimization are employed to analyze convergence properties of DRS, to tune the method and to deriv…
We show that the weak convergence of the Douglas--Rachford algorithm for finding a zero of the sum of two maximally monotone operators cannot be improved to strong convergence. Likewise, we show that strong convergence can fail for the method of partial inverses.
In this paper we give general recommendations for successful application of the Douglas-Rachford reflection method to convex and non-convex real matrix-completion problems. These guidelines are demonstrated by various illustrative examples.
We discuss recent positive experiences applying convex feasibility algorithms of Douglas--Rachford type to highly combinatorial and far from convex problems.
In this expository paper, we show how to use the Douglas-Rachford algorithm as a successful heuristic for finding magic squares. The Douglas-Rachford algorithm is an iterative projection method for solving feasibility problems. Although its convergence is only guaranteed in the convex setting, the algorithm has been successfully applied to a number of similar nonconvex problems, such as solving Sudoku puzzles. We present two formulations of the nonconvex feasibility problem of finding magic squares, which are inspired by those of Sudoku, and test the Douglas-Rachford algorithm on them.
Using the construction of a Lyapunov function, it is shown that the Douglas-Rachford iteration with respect to a sphere and a line in $\mathbb R^d$ is robustly $\mathcal{KL}$-stable. This implies a convergence which is stronger than uniform convergence on compact sets.
The Douglas-Rachford algorithm is widely used in sparse signal processing for minimizing a sum of two convex functions. In this paper, we consider the case where one of the functions is weakly convex but the other is strongly convex so that the sum is convex. We provide a condition that ensures the convergence of the same Douglas-Rachford iterations, provided that the strongly convex function is smooth. We also present a modified Douglas-Rachford algorithm that does not impose a smoothness condition for the convex function. We then provide a discussion on the convergence speed of the two types…
It is shown that away from the origin, the Douglas-Rachford operator with respect to a sphere and a convex set in a Hilbert space can be approximated by a another operator which satisfies a weak ergodic theorem. Similar results for other projection and reflection operators are also discussed.
We analyse the behaviour of the newly introduced cyclic Douglas-Rachford algorithm for finding a point in the intersection of a finite number of closed convex sets. This work considers the case in which the target intersection set is possibly empty.
We propose a splitting method for solving an equilibrium problem involving the sum of two bifunctions satisfying standard conditions. We prove that this problem is equivalent to find a zero of two appropriate maximally monotone operators. Our algorithm is a consequence of the Douglas--Rachford splitting applied to this auxiliary monotone inclusion. Connections between monotone inclusions and equilibrium problems are studied.
In recent years, the Douglas-Rachford splitting method has been shown to be effective at solving many non-convex optimization problems. In this paper we present a local convergence analysis for non-convex feasibility problems and show that both finite termination and local linear convergence are obtained. For a generalization of the Sudoku puzzle, we prove that the local linear rate of convergence of Douglas-Rachford is exactly $\frac{\sqrt{5}}{5}$ and independent of puzzle size. For the $s$-queens problem we prove that Douglas-Rachford converges after a finite number of iterations. Numerical …
In this paper we present two Douglas-Rachford inspired iteration schemes which can be applied directly to N-set convex feasibility problems in Hilbert space. Our main results are weak convergence of the methods to a point whose nearest point projections onto each of the N sets coincide. For affine subspaces, convergence is in norm. Initial results from numerical experiments, comparing our methods to the classical (product-space) Douglas-Rachford scheme, are promising.
In this paper, we study the generalized Douglas-Rachford algorithm and its cyclic variants which include many projection-type methods such as the classical Douglas-Rachford algorithm and the alternating projection algorithm. Specifically, we establish several local linear convergence results for the algorithm in solving feasibility problems with finitely many closed possibly nonconvex sets under different assumptions. Our findings not only relax some regularity conditions but also improve linear convergence rates in the literature. In the presence of convexity, the linear convergence is global…
We establish a region of convergence for the proto-typical non-convex Douglas-Rachford iteration which finds a point on the intersection of a line and a circle. Previous work on the non-convex iteration [2] was only able to establish local convergence, and was ineffective in that no explicit region of convergence could be given.
The Douglas--Rachford algorithm is a classical and very successful splitting method for finding the zeros of the sums of monotone operators. When the underlying operators are normal cone operators, the algorithm solves a convex feasibility problem. In this paper, we provide a detailed study of the Douglas--Rachford iterates and the corresponding {shadow sequence} when the sets are affine subspaces that do not necessarily intersect. We prove strong convergence of the shadows to the nearest generalized solution. Our results extend recent work from the consistent to the inconsistent case. Various…
We study the behavior of the Douglas-Rachford algorithm on the graph vertex-coloring problem. Given a graph and a number of colors, the goal is to find a coloring of the vertices so that all adjacent vertex pairs have different colors. In spite of the combinatorial nature of this problem, the Douglas-Rachford algorithm was recently shown to be a successful heuristic for solving a wide variety of graph coloring instances, when the problem was cast as a feasibility problem on binary indicator variables. In this work we consider a different formulation, based on semidefinite programming. The much…
In this paper, we propose and study the iteration complexity of an inexact Douglas-Rachford splitting (DRS) method and a Douglas-Rachford-Tseng's forward-backward (F-B) splitting method for solving two-operator and four-operator monotone inclusions, respectively. The former method (although based on a slightly different mechanism of iteration) is motivated by the recent work of J. Eckstein and W. Yao, in which an inexact DRS method is derived from a special instance of the hybrid proximal extragradient (HPE) method of Solodov and Svaiter, while the latter one combines the proposed inexact DRS …
The Douglas-Rachford algorithm can be represented as the fixed point iteration of a firmly nonexpansive operator. When the operator has no fixed points, the algorithm's iterates diverge, but the difference between consecutive iterates converges to the so-called minimal displacement vector, which can be used to certify infeasibility of an optimization problem. In this paper, we establish new properties of the minimal displacement vector, which allow us to generalize some existing results.
The Douglas-Rachford algorithm is one of the most prominent splitting algorithms for solving convex optimization problems. Recently, the method has been successful in finding a generalized solution (provided that one exists) for optimization problems in the inconsistent case, i.e., when a solution does not exist. The convergence analysis of the inconsistent case hinges on the study of the range of the displacement operator associated with the Douglas-Rachford splitting operator and the corresponding minimal displacement vector. In this paper, we provide a formula for the range of the Douglas-R…
We discuss the Douglas-Rachford algorithm to solve the feasibility problem for two closed sets $A,B$ in $\mathbb{R}^d$. We prove its local convergence to a fixed point when $A,B$ are finite unions of convex sets. We also show that for more general nonconvex sets the scheme may fail to converge and start to cycle, and may then even fail to solve the feasibility problem.
We consider the monotone inclusion problem with a sum of 3 operators, in which 2 are monotone and 1 is monotone-Lipschitz. The classical Douglas--Rachford and Forward-backward-forward methods respectively solve the monotone inclusion problem with a sum of 2 monotone operators and a sum of 1 monotone and 1 monotone-Lipschitz operators. We first present a method that naturally combines Douglas--Rachford and Forward-backward-forward and show that it solves the 3 operator problem under further assumptions, but fails in general. We then present a method that naturally combines Douglas--Rachford and…
Douglas-Rachford method is a splitting algorithm for finding a zero of the sum of two maximal monotone operators. Each of its iterations requires the sequential solution of two proximal subproblems. The aim of this work is to present a fully inexact version of Douglas-Rachford method wherein both proximal subproblems are solved approximately within a relative error tolerance. We also present a semi-inexact variant in which the first subproblem is solved exactly and the second one inexactly. We prove that both methods generate sequences weakly convergent to the solution of the underlying inclus…
In this paper we study new algorithmic structures with Douglas- Rachford (DR) operators to solve convex feasibility problems. We propose to embed the basic two-set-DR algorithmic operator into the String-Averaging Projections (SAP) and into the Block-Iterative Pro- jection (BIP) algorithmic structures, thereby creating new DR algo- rithmic schemes that include the recently proposed cyclic Douglas- Rachford algorithm and the averaged DR algorithm as special cases. We further propose and investigate a new multiple-set-DR algorithmic operator. Convergence of all these algorithmic schemes is studi…
The Douglas-Rachford projection algorithm is an iterative method used to find a point in the intersection of closed constraint sets. The algorithm has been experimentally observed to solve various nonconvex feasibility problems which current theory cannot sufficiently explain. In this paper, we prove convergence of the Douglas-Rachford algorithm in a potentially nonconvex setting. Our analysis relies on the existence of a Lyapunov-type functional whose convexity properties are not tantamount to convexity of the original constraint sets. Moreover, we provide various nonconvex examples in which …
Recently, several convergence rate results for Douglas-Rachford splitting and the alternating direction method of multipliers (ADMM) have been presented in the literature. In this paper, we show global linear convergence rate bounds for Douglas-Rachford splitting and ADMM under strong convexity and smoothness assumptions. We further show that the rate bounds are tight for the class of problems under consideration for all feasible algorithm parameters. For problems that satisfy the assumptions, we show how to select step-size and metric for the algorithm that optimize the derived convergence ra…
This paper derives new inexact variants of the Douglas-Rachford splitting method for maximal monotone operators and the alternating direction method of multipliers (ADMM) for convex optimization. The analysis is based on a new inexact version of the proximal point algorithm that includes both an inertial step and overrelaxation. We apply our new inexact ADMM method to LASSO and logistic regression problems and obtain somewhat better computational performance than earlier inexact ADMM methods.
We present the Douglas-Rachford algorithm as a successful heuristic for solving graph coloring problems. Given a set of colors, these type of problems consist in assigning a color to each node of a graph, in such a way that every pair of adjacent nodes are assigned with different colors. We formulate the graph coloring problem as an appropriate feasibility problem that can be effectively solved by the Douglas-Rachford algorithm, despite the nonconvexity arising from the combinatorial nature of the problem. Different modifications of the graph coloring problem and applications are also presente…
Based on a degenerate proximal point analysis, we show that the Douglas-Rachford splitting can be reduced to a well-defined resolvent, but generally fails to be a proximal mapping. This extends the recent result of [Bauschke, Schaad and Wang. Math. Program., 168 (2018), pp. 55-61] to more general setting. The related concepts and consequences are also discussed. In particular, the results regarding the maximal and cyclic monotonicity are instrumental for analyzing many operator splitting algorithms.
Recently, heuristics based on the Douglas-Rachford splitting algorithm and the alternating direction method of multipliers (ADMM) have found empirical success in minimizing convex functions over nonconvex sets, but not much has been done to improve the theoretical understanding of them. In this paper, we investigate convergence of these heuristics. First, we characterize optimal solutions of minimization problems involving convex cost functions over nonconvex constraint sets. We show that these optimal solutions are related to the fixed point set of the underlying nonconvex Douglas-Rachford op…
The Douglas-Rachford algorithm is a classical and very successful method for solving optimization and feasibility problems. In this paper, we provide novel conditions sufficient for finite convergence in the context of convex feasibility problems. Our analysis builds upon, and considerably extends, pioneering work by Spingarn. Specifically, we obtain finite convergence in the presence of Slater's condition in the affine-polyhedral and in a hyperplanar-epigraphical case. Various examples illustrate our results. Numerical experiments demonstrate the competitiveness of the Douglas-Rachford algori…
In this paper, we present a method for identifying infeasible, unbounded, and pathological conic programs based on Douglas-Rachford splitting, or equivalently ADMM. When an optimization program is infeasible, unbounded, or pathological, the iterates of Douglas-Rachford splitting diverge. Somewhat surprisingly, such divergent iterates still provide useful information, which our method uses for identification. In addition, for strongly infeasible problems the method produces a separating hyperplane and informs the user on how to minimally modify the given problem to achieve strong feasibility. A…
In this work, we propose a new algorithm for finding a zero in the sum of two monotone operators where one is assumed to be single-valued and Lipschitz continuous. This algorithm naturally arises from a non-standard discretization of a continuous dynamical system associated with the Douglas--Rachford splitting algorithm. More precisely, it is obtained by performing an explicit, rather than implicit, discretization with respect to one of the operators involved. Each iteration of the proposed algorithm requires the evaluation of one forward and one backward operator.
Douglas-Rachford splitting and the alternating direction method of multipliers (ADMM) can be used to solve convex optimization problems that consist of a sum of two functions. Convergence rate estimates for these algorithms have received much attention lately. In particular, linear convergence rates have been shown by several authors under various assumptions. One such set of assumptions is strong convexity and smoothness of one of the functions in the minimization problem. The authors recently provided a linear convergence rate bound for such problems. In this paper, we show that this rate bo…
The basic optimization problem of road design is quite challenging due to a objective function that is the sum of nonsmooth functions and the presence of set constraints. In this paper, we model and solve this problem by employing the Douglas-Rachford splitting algorithm. This requires a careful study of new proximity operators related to minimizing area and to the stadium norm. We compare our algorithm to a state-of-the-art projection algorithm. Our numerical results illustrate the potential of this algorithm to significantly reduce cost in road design.
Although originally designed and analyzed for convex problems, the alternating direction method of multipliers (ADMM) and its close relatives, Douglas-Rachford splitting (DRS) and Peaceman-Rachford splitting (PRS), have been observed to perform remarkably well when applied to certain classes of structured nonconvex optimization problems. However, partial global convergence results in the nonconvex setting have only recently emerged. In this paper we show how the Douglas-Rachford envelope (DRE), introduced in 2014, can be employed to unify and considerably simplify the theory for devising globa…
Douglas-Rachford method is a splitting algorithm for finding a zero of the sum of two maximal monotone operators. Weak convergence in this method to a solution of the underlying monotone inclusion problem in the general case remained an open problem for 30 years and was prove by the author 7 year ago. The proof presented at that occasion was cluttered with technicalities because we considered the inexact version with summable errors. The aim of this note is to present a streamlined proof of this result.
The Douglas Rachford algorithm is an algorithm that converges to a minimizer of a sum of two convex functions. The algorithm consists in fixed point iterations involving computations of the proximity operators of the two functions separately. The paper investigates a stochastic version of the algorithm where both functions are random and the step size is constant. We establish that the iterates of the algorithm stay close to the set of solution with high probability when the step size is small enough. Application to structured regularization is considered.
More than 40 years ago, Lions and Mercier introduced in a seminal paper the Douglas-Rachford algorithm. Today, this method is well recognized as a classical and highly successful splitting method to find minimizers of the sum of two (not necessarily smooth) convex functions. While the underlying theory has matured, one case remains a mystery: the behaviour of the shadow sequence when the given functions have disjoint domains. Building on previous work, we establish for the first time weak and value convergence of the shadow sequence generated by the Douglas-Rachford algorithm in a setting of u…
