We leverage the connections between nonexpansive maps, monotone Lipschitz operators, and proximal mappings to obtain near-optimal (i.e., optimal up to poly-log factors in terms of iteration complexity) and parameter-free methods for solving monotone inclusion problems. These results immediately translate into near-optimal guarantees for approximating strong solutions to variational inequality problems, approximating convex-concave min-max optimization problems, and minimizing the norm of the gradient in min-max optimization problems. Our analysis is based on a novel and simple potential-based proof of convergence of Halpern iteration, a classical iteration for finding fixed points of nonexpansive maps. Additionally, we provide a series of algorithmic reductions that highlight connections between different problem classes and lead to lower bounds that certify near-optimality of the studied methods.
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Pontificia Universidad Católica de Chile（智利天主教大学）
We study stochastic monotone inclusion problems, which widely appear in machine learning applications, including robust regression and adversarial learning. We propose novel variants of stochastic Halpern iteration with recursive variance reduction. In the cocoercive -- and more generally Lipschitz-monotone -- setup, our algorithm attains $ε$ norm of the operator with $\mathcal{O}(\frac{1}{ε^3})$ stochastic operator evaluations, which significantly improves over state of the art $\mathcal{O}(\frac{1}{ε^4})$ stochastic operator evaluations required for existing monotone inclusion solvers applie…
In this paper, we develop a new type of accelerated algorithms to solve some classes of maximally monotone equations as well as monotone inclusions. Instead of using Nesterov's accelerating approach, our methods rely on a so-called Halpern-type fixed-point iteration in [32], and recently exploited by a number of researchers, including [24, 70]. Firstly, we derive a new variant of the anchored extra-gradient scheme in [70] based on Popov's past extra-gradient method to solve a maximally monotone equation $G(x) = 0$. We show that our method achieves the same $\mathcal{O}(1/k)$ convergence rate (…
An Inexact Halpern Iteration with Application to Distributionally Robust Optimization
University of Maryland College Park（马里兰大学学院公园分校）；Weierstrass Institute for Applied Analysis and Stochastics Berlin, Germany（柏林应用分析与随机学研究所）；National University of Singapore（新加坡国立大学）
University of Vienna（维也纳大学）；University of Southampton（南安普顿大学）
We introduce a relaxed inertial forward-backward-forward (RIFBF) splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitz continuous operator. This work aims to extend Tseng's forward-backward-forward method by both using inertial effects as well as relaxation parameters. We formulate first a second order dynamical system which approaches the solution set of the monotone inclusion problem to be solved and provide an asymptotic analysis for its trajectories. We provide for RIFBF, which follows by explicit time di…
In this paper we provide a splitting method for finding a zero of the sum of a maximally monotone operator, a lipschitzian monotone operator, and a normal cone to a closed vectorial subspace of a real Hilbert space. The problem is characterized by a simpler monotone inclusion involving only two operators: the partial inverse of the maximally monotone operator with respect to the vectorial subspace and a suitable lipschitzian monotone operator. By applying the Tseng's method in this context we obtain a splitting algorithm that exploits the whole structure of the original problem and generalizes…
Monotone inclusions involving the sum of three maximally monotone operators or more have received much attention in recent years. In this paper, we propose three splitting algorithms for finding a zero of the sum of four monotone operators, which are two maximally monotone operators, one monotone Lipschitz operator, and one cocoercive operator. These three splitting algorithms are based on the forward-reflected-Douglas-Rachford splitting algorithm, backward-forward-reflected-backward splitting algorithm, and backward-reflected-forward-backward splitting algorithm, respectively. As applications…
We consider the problem of solving dual monotone inclusions involving sums of composite parallel-sum type operators. A feature of this work is to exploit explicitly the cocoercivity of some of the operators appearing in the model. Several splitting algorithms recently proposed in the literature are recovered as special cases.
The principle underlying this paper is the basic observation that the problem of simultaneously solving a large class of composite monotone inclusions and their duals can be reduced to that of finding a zero of the sum of a maximally monotone operator and a linear skew-adjoint operator. An algorithmic framework is developed for solving this generic problem in a Hilbert space setting. New primal-dual splitting algorithms are derived from this framework for inclusions involving composite monotone operators, and convergence results are established. These algorithms draw their simplicity and effic…
Kuhn-Tucker points play a fundamental role in the analysis and the numerical solution of monotone inclusion problems, providing in particular both primal and dual solutions. We propose a class of strongly convergent algorithms for constructing the best approximation to a reference point from the set of Kuhn-Tucker points of a general Hilbertian composite monotone inclusion problem. Applications to systems of coupled monotone inclusions are presented. Our framework does not impose additional assumptions on the operators present in the formulation, and it does not require knowledge of the norm o…
We focus on the linear convergence of generalized proximal point algorithms for solving monotone inclusion problems. Under the assumption that the associated monotone operator is metrically subregular or that the inverse of the monotone operator is Lipschitz continuous, we provide Q-linear and R-linear convergence results on generalized proximal point algorithms. Comparisons between our results and related ones in the literature are presented in remarks of this work.
We propose a splitting algorithm for solving a system of composite monotone inclusions formulated in the form of the extended set of solutions in real Hilbert spaces. The resluting algorithm is a an extension of the algorithm in [4]. The weak convergence of the algorithm proposed is proved. Applications to minimization problems is demonstrated.
We give criteria for weak and strong invariant closed sets for differential inclusions given in $\mathbb{R}^{n}$ and governed by Lipschitz Cusco perturbations of maximal monotone operators. Correspondingly, we provide different characterizations for the\ associated strong Lyapunov functions. The resulting conditions only depend on the data of the system.
We introduce the first operator splitting method for composite monotone inclusions outside of Hilbert spaces. The proposed primal-dual method constructs iteratively the best Bregman approximation to an arbitrary point from the Kuhn-Tucker set of a composite monotone inclusion. Strong convergence is established in reflexive Banach spaces without requiring additional restrictions on the monotone operators or knowledge of the norms of the linear operators involved in the model. The monotone operators are activated via Bregman distance-based resolvent operators. The method is novel even in Euclide…
We propose a variable metric forward-backward splitting algorithm and prove its convergence in real Hilbert spaces. We then use this framework to derive primal-dual splitting algorithms for solving various classes of monotone inclusions in duality. Some of these algorithms are new even when specialized to the fixed metric case. Various applications are discussed.
The aim of this article is to present two different primal-dual methods for solving structured monotone inclusions involving parallel sums of compositions of maximally monotone operators with linear bounded operators. By employing some elaborated splitting techniques, all of the operators occurring in the problem formulation are processed individually via forward or backward steps. The treatment of parallel sums of linearly composed maximally monotone operators is motivated by applications in imaging which involve first- and second-order total variation functionals, to which a special attentio…
We propose an inertial forward-backward splitting algorithm to compute the zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal-dual splitting methods for solving structured systems of composite monotone inclusions and prove their convergence. Our results extend to the stochastic and inertial setting various types of structure…
Monotone inclusions have wide applications in solving various convex optimization problems arising in signal and image processing, machine learning, and medical image reconstruction. In this paper, we propose a new splitting algorithm for finding a zero of the sum of a maximally monotone operator, a monotone Lipschitzian operator, and a cocoercive operator, which is called outer reflected forward-backward splitting algorithm. Under mild conditions on the iterative parameters, we prove the convergence of the proposed algorithm. As applications, we employ the proposed algorithm to solve composit…
In this paper, based on a double inertial extrapolation steps strategy and relaxation techniques, we introduce a new Tseng splitting method with double inertial extrapolation steps and self-adaptive step sizes for solving monotone inclusion problems in real Hilbert spaces. Finally, several numerical experiments are provided to illustrate the performance and theoretical outcomes of our algorithm.
In this paper, we introduce for the first time a class of state-dependent maximal monotone differential inclusions. Then the existence and uniqueness of solutions are obtained by using an implicit discretization scheme and a kind of hypo-monotonicity assumption respectively. In addition, a characterization for nonsmooth Lyapunov pairs associated with such systems is provided. Our result can be applied to study state-dependent sweeping processes and Lur'e dynamical systems. It is new even the involved maximal monotone operators depend only on the time.
Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems
Toulouse School of Economics（图卢兹经济学院）；University of Toulouse（图卢兹大学）；IRIT（图卢兹计算机科学研究所）；CNRS（法国国家科学研究中心）；Université Toulouse III Paul Sabatier（图卢兹第三大学保罗·萨巴蒂埃大学）；Institut Universitaire de France（法兰西大学研究院）；CVN（认知与神经计算研究中心）；CentraleSupélec（巴黎中央理工-高等电力学院）；Université Paris-Saclay（巴黎萨克雷大学）
The proximal extrapolated gradient method \cite{Malitsky18a} is an extension of the projected reflected gradient method \cite{Malitsky15}. Both methods were proposed for solving the classic variational inequalities. In this paper, we investigate the projected reflected gradient method, in the general setting, for solving monotone inclusions involving Lipschitzian operators. As a result, we obtain a simple method for finding a zero point of the sum of two monotone operators where one of them is Lipschizian. We also show that one can improve the range of the stepsize of this method for the case …
A general primal-dual splitting algorithm for solving systems of structured coupled monotone inclusions in Hilbert spaces is introduced and its asymptotic behavior is analyzed. Each inclusion in the primal system features compositions with linear operators, parallel sums, and Lipschitzian operators. All the operators involved in this structured model are used separately in the proposed algorithm, most steps of which can be executed in parallel. This provides a flexible solution method applicable to a variety of problems beyond the reach of the state-of-the-art. Several applications are discuss…
A parallel splitting method is proposed for solving systems of coupled monotone inclusions in Hilbert spaces. Convergence is established for a wide class of coupling schemes. Unlike classical alternating algorithms, which are limited to two variables and linear coupling, our parallel method can handle an arbitrary number of variables as well as nonlinear coupling schemes. The breadth and flexibility of the proposed framework is illustrated through applications in the areas of evolution inclusions, dynamical games, signal recovery, image decomposition, best approximation, network flows, and var…
We propose a primal-dual splitting algorithm for solving monotone inclusions involving a mixture of sums, linear compositions, and parallel sums of set-valued and Lipschitzian operators. An important feature of the algorithm is that the Lipschitzian operators present in the formulation can be processed individually via explicit steps, while the set-valued operators are processed individually via their resolvents. In addition, the algorithm is highly parallel in that most of its steps can be executed simultaneously. This work brings together and notably extends various types of structured monot…
We introduce a generalized forward-backward splitting method with penalty term for solving monotone inclusion problems involving the sum of a finite number of maximally monotone operators and the normal cone to the nonempty set of zeros of another maximal monotone operator. We show weak ergodic convergence of the generated sequence of iterates to a solution of the considered monotone inclusion problem, provided the condition corresponded to the Fitzpatrick function of the operator describing the set of the normal cone is fulfilled. Under strong monotonicity of an operator, we show strong conve…
In this paper, we propose a randomized intertial block-coordinate primaldual fixed point algorithm to solve a wide array of monotone inclusion problems base on the modification of the heavy ball method of Nesterov. These methods rely on a sweep of blocks of variables which are activated at each iteration according to a random rule. To this end we formulate the inertial version of the Krasnosel'skii-Mann algorithm for approximating the set of fixed points of a quasinonexpansive operator, for which we also provide an exhaustive convergence analysis. As a by-product, we can obtain some intertial …
This paper studies the iteration-complexity of new regularized hybrid proximal extragradient (HPE)-type methods for solving monotone inclusion problems (MIPs). The new (regularized HPE-type) methods essentially consist of instances of the standard HPE method applied to regularizations of the original MIP. It is shown that its pointwise iteration-complexity considerably improves the one of the HPE method while approaches (up to a logarithmic factor) the ergodic iteration-complexity of the latter method.
In this paper, we propose a stochastic version of the classical Tseng's forward-backward-forward method with inertial term for solving monotone inclusions given by the sum of a maximal monotone operator and a single-valued monotone operator in real Hilbert spaces. We obtain the almost sure convergence for the general case and the rate $\mathcal{O}(1/n)$ in expectation for the strong monotone case. Furthermore, we derive $\mathcal{O}(1/n)$ rate convergence of the primal-dual gap for saddle point problems.
We provide two weakly convergent algorithms for finding a zero of the sum of a maximally monotone operator, a cocoercive operator, and the normal cone to a closed vector subspace of a real Hilbert space. The methods exploit the intrinsic structure of the problem by activating explicitly the cocoercive operator in the first step, and taking advantage of a vector space decomposition in the second step. The second step of the first method is a Douglas-Rachford iteration involving the maximally monotone operator and the normal cone. In the second method it is a proximal step involving the partial …
Resolvents of set-valued operators play a central role in various branches of mathematics and in particular in the design and the analysis of splitting algorithms for solving monotone inclusions. We propose a generalization of this notion, called warped resolvent, which is constructed with the help of an auxiliary operator. The properties of warped resolvents are investigated and connections are made with existing notions. Abstract weak and strong convergence principles based on warped resolvents are proposed and shown to not only provide a synthetic view of splitting algorithms but to also co…
We propose and analyze a new dynamical system with a closed-loop control law in a Hilbert space $\mathcal{H}$, aiming to shed light on the acceleration phenomenon for \textit{monotone inclusion} problems, which unifies a broad class of optimization, saddle point and variational inequality (VI) problems under a single framework. Given $A: \mathcal{H} \rightrightarrows \mathcal{H}$ that is maximal monotone, we propose a closed-loop control system that is governed by the operator $I - (I + λ(t)A)^{-1}$, where a feedback law $λ(\cdot)$ is tuned by the resolution of the algebraic equation $λ(t)\|(I…
In this paper we provide a splitting algorithm for solving coupled monotone inclusions in a real Hilbert space involving the sum of a normal cone to a vector subspace, a maximally monotone, a monotone-Lipschitzian, and a cocoercive operator. The proposed method takes advantage of the intrinsic properties of each operator and generalizes the method of partial inverses and the forward-backward-half forward splitting, among other methods. At each iteration, our algorithm needs two computations of the Lipschitzian operator while the cocoercive operator is activated only once. By using product spac…
Differential inclusions with compact, upper semi-continuous, not necessarily convex right-hand sides in R^n are studied. Under a weakened monotonicity-type condition the existence of solutions is proved.
In this work, we study resolvent splitting algorithms for solving composite monotone inclusion problems. The objective of these general problems is finding a zero in the sum of maximally monotone operators composed with linear operators. Our main contribution is establishing the first primal-dual splitting algorithm for composite monotone inclusions with minimal lifting. Specifically, the proposed scheme reduces the dimension of the product space where the underlying fixed point operator is defined, in comparison to other algorithms, without requiring additional evaluations of the resolvent op…
We deal with monotone inclusion problems of the form $0\in Ax+Dx+N_C(x)$ in real Hilbert spaces, where $A$ is a maximally monotone operator, $D$ a cocoercive operator and $C$ the nonempty set of zeros of another cocoercive operator. We propose a forward-backward penalty algorithm for solving this problem which extends the one proposed by H. Attouch, M.-O. Czarnecki and J. Peypouquet in [3]. The condition which guarantees the weak ergodic convergence of the sequence of iterates generated by the proposed scheme is formulated by means of the Fitzpatrick function associated to the maximally monoto…
We propose a new class of primal-dual Fejer monotone algorithms for solving systems of com- posite monotone inclusions. Our construction is inspired by a framework used by Eckstein and Svaiter for the basic problem of finding a zero of the sum of two monotone operators. At each iteration, points in the graph of the monotone operators present in the model are used to con- struct a half-space containing the Kuhn-Tucker set associated with the system. The primal-dual update is then obtained via a relaxed projection of the current iterate onto this half-space. An important feature that distinguish…
In this work, we propose a simple modification of the forward-backward splitting method for finding a zero in the sum of two monotone operators. Our method converges under the same assumptions as Tseng's forward-backward-forward method, namely, it does not require cocoercivity of the single-valued operator. Moreover, each iteration only requires one forward evaluation rather than two as is the case for Tseng's method. Variants of the method incorporating a linesearch, relaxation and inertia, or a structured three operator inclusion are also discussed.
In this paper, based on inertial and Tseng's ideas, we propose two projection-based algorithms to solve a monotone inclusion problem in infinite dimensional Hilbert spaces. Solution theorems of strong convergence are obtained under the certain conditions. Some numerical experiments are presented to illustrate that our algorithms are efficient than the existing results.
The aim of this survey is to present the main important techniques and tools from variational analysis used for first and second order dynamical systems of implicit type for solving monotone inclusions and non-smooth optimization problems. The differential equations are expressed by means of the resolvent (in case of a maximally monotone set valued operator) or the proximal operator for non-smooth functions. The asymptotic analysis of the trajectories generated relies on Lyapunov theory, where the appropriate energy functional plays a decisive role. While the most part of the paper is related …
In this work we propose a new splitting technique, namely Asymmetric Forward-Backward-Adjoint splitting, for solving monotone inclusions involving three terms, a maximally monotone, a cocoercive and a bounded linear operator. Classical operator splitting methods, like Douglas-Rachford and Forward-Backward splitting are special cases of our new algorithm. Asymmetric Forward-Backward-Adjoint splitting unifies, extends and sheds light on the connections between many seemingly unrelated primal-dual algorithms for solving structured convex optimization problems proposed in recent years. More import…
In this paper, we study the existence and the stability in the sense of Lyapunov of solutions for\ differential inclusions governed by the normal cone to a prox-regular set and subject to a Lipschitzian perturbation. We prove that such, apparently, more general nonsmooth dynamics can be indeed remodelled into the classical theory of differential inclusions involving maximal monotone operators. This result is new in the literature and permits us to make use of the rich and abundant achievements in this class of monotone operators to derive the desired existence result and stability analysis, as…
In this work, we propose and analyse two splitting algorithms for finding a zero of the sum of three monotone operators, one of which is assumed to be Lipschitz continuous. Each iteration of these algorithms require one forward evaluation of the Lipschitz continuous operator and one resolvent evaluation of each of the other two operators. By specialising to two operator inclusions, we recover the forward-reflected-backward and the reflected-forward-backward splitting methods as particular cases. The inspiration for the proposed algorithms arises from interpretations of the aforementioned refle…
We propose a new primal-dual splitting method for solving composite inclusions involving Lipschitzian, and parallel-sum-type monotone operators. Our approach extends the framework in \cite{Siopt4} to a more general class of monotone inclusions in a nontrivial fashion. The main idea is to represent the solution set of both the primal and dual problems using their associated Kuhn-Tucker set, and then develop a projected method to successively approximate a feasible point of the Kuhn-Tucker set. We propose a splitting algorithm based on the resolvent of each maximally monotone operator to constru…
In this paper we propose a resolvent splitting with minimal lifting for finding a zero of the sum of $n\ge 2$ maximally monotone operators involving the composition with a linear bounded operator. The resolvent of each monotone operator, the linear operator, and its adjoint are computed exactly once in the proposed algorithm. In the case when the linear operator is the identity, we recover the resolvent splitting with minimal lifting developed in Malitsky-Tam (2021). We also derive a new resolvent splitting for solving the composite monotone inclusion in the case $n=2$ with minimal $1-$fold li…
