The three-operators splitting algorithm is a popular operator splitting method for finding the zeros of the sum of three maximally monotone operators, with one of which is cocoercive operator. In this paper, we propose a class of inertial three-operator splitting algorithm. The convergence of the proposed algorithm is proved by applying the inertial Krasnoselskii-Mann iteration under certain conditions on the iterative parameters in real Hilbert spaces. As applications, we develop an inertial three-operator splitting algorithm to solve the convex minimization problem of the sum of three convex functions, where one of them is differentiable with Lipschitz continuous gradient. Finally, we conduct numerical experiments on a constrained image inpainting problem with nuclear norm regularization. Numerical results demonstrate the advantage of the proposed inertial three-operator splitting algorithms.
In this paper, based on a double inertial extrapolation steps strategy and relaxation techniques, we introduce a new Tseng splitting method with double inertial extrapolation steps and self-adaptive step sizes for solving monotone inclusion problems in real Hilbert spaces. Finally, several numerical experiments are provided to illustrate the performance and theoretical outcomes of our algorithm.
We propose an inertial forward-backward splitting algorithm to compute the zero of a sum of two monotone operators allowing for stochastic errors in the computation of the operators. More precisely, we establish almost sure convergence in real Hilbert spaces of the sequence of iterates to an optimal solution. Then, based on this analysis, we introduce two new classes of stochastic inertial primal-dual splitting methods for solving structured systems of composite monotone inclusions and prove their convergence. Our results extend to the stochastic and inertial setting various types of structure…
We introduce an inertial variant of the forward-Douglas-Rachford splitting and analyze its convergence. We specify an instance of the proposed method to the three-composite convex minimization template. We provide practical guidance on the selection of the inertial parameter based on the adaptive starting idea. Finally, we illustrate the practical performance of our method in various machine learning applications.
This paper derives new inexact variants of the Douglas-Rachford splitting method for maximal monotone operators and the alternating direction method of multipliers (ADMM) for convex optimization. The analysis is based on a new inexact version of the proximal point algorithm that includes both an inertial step and overrelaxation. We apply our new inexact ADMM method to LASSO and logistic regression problems and obtain somewhat better computational performance than earlier inexact ADMM methods.
We study the convergence properties of a general inertial first-order proximal splitting algorithm for solving nonconvex nonsmooth optimization problems. Using the Kurdyka--Łojaziewicz (KL) inequality we establish new convergence rates which apply to several inertial algorithms in the literature. Our basic assumption is that the objective function is semialgebraic, which lends our results broad applicability in the fields of signal processing and machine learning. The convergence rates depend on the exponent of the "desingularizing function" arising in the KL inequality. Depending on this expo…
For solving structured monotone inclusion problems involving the sum of finitely many maximal monotone operators, we propose and study a relative-error inertial-relaxed inexact projective splitting algorithm. The proposed algorithm benefits from a combination of inertial and relaxation effects, which are both controlled by parameters within a certain range. We propose sufficient conditions on these parameters and study the interplay between them in order to guarantee weak convergence of sequences generated by our algorithm. Additionally, the proposed algorithm also benefits from inexact subpro…
We study inertial versions of primal-dual proximal splitting, also known as the Chambolle--Pock method. Our starting point is the preconditioned proximal point formulation of this method. By adding correctors corresponding to the anti-symmetric part of the relevant monotone operator, using a FISTA-style gap unrolling argument, we are able to derive gap estimates instead of merely ergodic gap estimates. Moreover, based on adding a diagonal component to this corrector, we are able to combine strong convexity based acceleration with inertial acceleration. We test our proposed method on image proc…
In this paper we propose new numerical algorithms in the setting of unconstrained optimization problems and we study the rate of convergence in the iterates of the objective function. Furthermore, our algorithms are based upon splitting and symplectic methods and they preserve the energy properties of the inherent continuous dynamical system that contains a Hessian perturbation. At the same time, we show that Nesterov gradient method is equivalent to a Lie-Trotter splitting applied to a Hessian driven damping system. Finally, some numerical experiments are presented in order to validate the th…
In this paper, we propose a multi-step inertial Forward--Backward splitting algorithm for minimizing the sum of two non-necessarily convex functions, one of which is proper lower semi-continuous while the other is differentiable with a Lipschitz continuous gradient. We first prove global convergence of the scheme with the help of the Kurdyka-Łojasiewicz property. Then, when the non-smooth part is also partly smooth relative to a smooth submanifold, we establish finite identification of the latter and provide sharp local linear convergence analysis. The proposed method is illustrated on a few p…
We consider an inertial primal-dual algorithm to compute the minimizations of the sum of two convex functions and the composition of another convex function with a continuous linear operator. With the idea of coordinate descent, we design a stochastic coordinate descent inertial primal-dual splitting algorithm. Moreover, in order to prove the convergence of the proposed inertial algorithm, we formulate first the inertial version of the randomized Krasnosel'skii-Mann iterations algorithm for approximating the set of fixed points of a nonexpansive operator and investigate its convergence propert…
We introduce and investigate the convergence properties of an inertial forward-backward-forward splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitzian operator. By making use of the product space approach, we expand it to the solving of inclusion problems involving mixtures of linearly composed and parallel-sum type monotone operators. We obtain in this way an inertial forward-backward-forward primal-dual splitting algorithm having as main characteristic the fact that in the iterative scheme all operators a…
A Double Inertial Forward-Backward Splitting Algorithm With Applications to Regression and Classification Problems
Department of Mathematics, Faculty of Science, Erzurum Technical University（数学系，科学学院，埃兹朱尔姆技术大学）；Department of Computer Engineering, Naval Academy National Defense University（计算机工程系，海军学院国防大学）
We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator $T$ and a single-valued monotone, Lipschitz continuous, and expectation-valued operator $V$. We draw motivation from the seminal work by Attouch and Cabot on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward-backward-forward (RISFBF) method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that (RISFBF) produces a sequence that weakly converges to the solution set…
We propose and study a novel stochastic inertial primal-dual approach to solve composite optimization problems. These latter problems arise naturally when learning with penalized regularization schemes. Our analysis provide convergence results in a general setting, that allows to analyze in a unified framework a variety of special cases of interest. Key in our analysis is considering the framework of splitting algorithm for solving a monotone inclusions in suitable product spaces and for a specific choice of preconditioning operators.
We consider an inertial primal-dual fixed point algorithm (IPDFP) to compute the minimizations of the following Problem (1.1). This is a full splitting approach, in the sense that the nonsmooth functions are processed individually via their proximity operators. The convergence of the IPDFP is obtained by reformulating the Problem (1.1) to the sum of three convex functions. This work brings together and notably extends several classical splitting schemes, like the primaldual method proposed by Chambolle and Pock, and the recent proximity algorithms of Charles A. et al designed for the L1/TV ima…
In this paper we propose a new operator splitting algorithm for distributed Nash equilibrium seeking under stochastic uncertainty, featuring relaxation and inertial effects. Our work is inspired by recent deterministic operator splitting methods, designed for solving structured monotone inclusion problems. The algorithm is derived from a forward-backward-forward scheme for solving structured monotone inclusion problems featuring a Lipschitz continuous and monotone game operator. To the best of our knowledge, this is the first distributed (generalized) Nash equilibrium seeking algorithm featuri…
The alternating direction method of multipliers (ADMM) is a popular method for solving convex separable minimization problems with linear equality constraints. The generalization of the two-block ADMM to the three-block ADMM is not trivial since the three-block ADMM is not convergence in general. Many variants of three-block ADMM have been developed with guarantee convergence. Besides the ADMM, the alternating minimization algorithm (AMA) is also an important algorithm for solving the convex separable minimization problem with linear equality constraints. The AMA is first proposed by Tseng, an…
The alternating direction method of multipliers (ADMM) is a widely used method for solving many convex minimization models arising in signal and image processing. In this paper, we propose an inertial ADMM for solving a two-block separable convex minimization problem with linear equality constraints. This algorithm is obtained by making use of the inertial Douglas-Rachford splitting algorithm to the corresponding dual of the primal problem. We study the convergence analysis of the proposed algorithm in infinite-dimensional Hilbert spaces. Furthermore, we apply the proposed algorithm on the rob…
Brandenburg Technical University Cottbus–Senftenberg（科特布斯-森夫滕堡勃兰登堡工业大学）；Saarland University（萨尔兰大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Tel Aviv University（特拉维夫大学）；University of Adelaide（阿德莱德大学）；LIX；École Polytechnique（巴黎综合理工学院）
Some recent methods for lossy signal and image compression store only a few selected pixels and fill in the missing structures by inpainting with a partial differential equation (PDE). Suitable operators include the Laplacian, the biharmonic operator, and edge-enhancing anisotropic diffusion (EED). The quality of such approaches depends substantially on the selection of the data that is kept. Optimising this data in the domain and codomain gives rise to challenging mathematical problems that shall be addressed in our work. In the 1D case, we prove results that provide insights into the difficu…
A local convergence result for abstract descent methods is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward--backward splitting method: iPiano---a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equival…
A problem of great interest in optimization is to minimize a sum of two closed, proper, and convex functions where one is smooth and the other has a computationally inexpensive proximal operator. In this paper we analyze a family of Inertial Forward-Backward Splitting (I-FBS) algorithms for solving this problem. We first apply a global Lyapunov analysis to I-FBS and prove weak convergence of the iterates to a minimizer in a real Hilbert space. We then show that the algorithms achieve local linear convergence for "sparse optimization", which is the important special case where the nonsmooth ter…
In this paper, we propose and study several strongly convergent versions of the forward-reflected-backward splitting method of Malitsky and Tam for finding a zero of the sum of two monotone operators in a real Hilbert space. Our proposed methods only require one forward evaluation of the single-valued operator and one backward evaluation of the set-valued operator at each iteration; a feature that is absent in many other available strongly convergent splitting methods in the literature. We also develop inertial versions of our methods and strong convergence results are obtained for these metho…
In this paper, based on inertial and Tseng's ideas, we propose two projection-based algorithms to solve a monotone inclusion problem in infinite dimensional Hilbert spaces. Solution theorems of strong convergence are obtained under the certain conditions. Some numerical experiments are presented to illustrate that our algorithms are efficient than the existing results.
University of Vienna（维也纳大学）；University of Southampton（南安普顿大学）
We introduce a relaxed inertial forward-backward-forward (RIFBF) splitting algorithm for approaching the set of zeros of the sum of a maximally monotone operator and a single-valued monotone and Lipschitz continuous operator. This work aims to extend Tseng's forward-backward-forward method by both using inertial effects as well as relaxation parameters. We formulate first a second order dynamical system which approaches the solution set of the monotone inclusion problem to be solved and provide an asymptotic analysis for its trajectories. We provide for RIFBF, which follows by explicit time di…
We propose and study a weakly convergent variant of the forward--backward algorithm for solving structured monotone inclusion problems. Our algorithm features a per-iteration deviation vector which provides additional degrees of freedom. The only requirement on the deviation vector to guarantee convergence is that its norm is bounded by a quantity that can be computed online. This approach provides great flexibility and opens up for the design of new and improved forward--backward-based algorithms, while retaining global convergence guarantees. These guarantees include linear convergence of ou…
TU Braunschweig（布伦瑞克工业大学）；Graz University of Technology（格拉茨工业大学）；AIT Austrian Institute of Technology GmbH（AIT奥地利技术研究所有限公司）
In this paper, we propose an inertial forward backward splitting algorithm to compute a zero of the sum of two monotone operators, with one of the two operators being co-coercive. The algorithm is inspired by the accelerated gradient method of Nesterov, but can be applied to a much larger class of problems including convex-concave saddle point problems and general monotone inclusions. We prove convergence of the algorithm in a Hilbert space setting and show that several recently proposed first-order methods can be obtained as special cases of the general algorithm. Numerical results show that …
University of Freiburg（弗莱堡大学）；BIOSS Centre for Biological Signalling Studies（BIOSS生物信号研究中心）；Graz University of Technology（格拉茨工业大学）
In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class of problems yields global convergence of the function values and the arguments. This makes the algorithm robust for usage on non-convex problems. The convergence result is obtained based on the \KL ine…
We address the generalized Nash equilibrium seeking problem for a population of agents playing aggregative games with affine coupling constraints. We focus on semi-decentralized communication architectures, where there is a central coordinator able to gather and broadcast signals of aggregative nature to the agents. By exploiting the framework of monotone operator theory and operator splitting, we first critically review the most relevant available algorithms and then design two novel schemes: (i) a single-layer, fixed-step algorithm with convergence guarantee for general (non cocoercive, non-…
University of Münster（明斯特大学）；University of Mannheim（曼海姆大学）
Variational methods in imaging are nowadays developing towards a quite universal and flexible tool, allowing for highly successful approaches on tasks like denoising, deblurring, inpainting, segmentation, super-resolution, disparity, and optical flow estimation. The overall structure of such approaches is of the form ${\cal D}(Ku) + α{\cal R} (u) \rightarrow \min_u$ ; where the functional ${\cal D}$ is a data fidelity term also depending on some input data $f$ and measuring the deviation of $Ku$ from such and ${\cal R}$ is a regularization functional. Moreover $K$ is a (often linear) forward o…
