We examine the linear convergence rates of variants of the proximal point method for finding zeros of maximal monotone operators. We begin by showing how metric subregularity is sufficient for linear convergence to a zero of a maximal monotone operator. This result is then generalized to obtain convergence rates for the problem of finding a common zero of multiple monotone operators by considering randomized and averaged proximal methods.
Based on the needs of convergence proofs of preconditioned proximal point methods, we introduce notions of partial strong submonotonicity and partial (metric) subregularity of set-valued maps. We study relationships between these two concepts, neither of which is generally weaker or stronger than the other one. For our algorithmic purposes, the novel submonotonicity turns out to be easier to employ than more conventional error bounds obtained from subregularity. Using strong submonotonicity, we demonstrate the linear convergence of the Primal-Dual Proximal splitting method to some strictly com…
In the setting of CAT(k) spaces, common fixed point iterations built from prox mappings (e.g. prox-prox, Krasnoselsky-Mann relaxations, nonlinear projected-gradients) converge locally linearly under the assumption of linear metric subregularity. Linear metric subregularity is in any case necessary for linearly convergent fixed point sequences, so the result is tight. To show this, we develop a theory of fixed point mappings that violate the usual assumptions of nonexpansiveness and firm nonexpansiveness in p-uniformly convex spaces.
Necessary and sufficient criteria for metric subregularity (or calmness) of set-valued mappings between general metric or Banach spaces are treated in the framework of the theory of error bounds for a special family of extended real-valued functions of two variables. A classification scheme for the general error bound and metric subregularity criteria is presented. The criteria are formulated in terms of several kinds of primal and subdifferential slopes.
In [2] we characterized in terms of a quadratic growth condition various metric regularity properties of the subdifferential of a lower semicontinuous convex function acting in a Hilbert space. Motivated by some recent results in [16] where the authors extend to Banach spaces the characterization of the strong regularity, we extend as well the characterizations for the metric subregularity and the strong subregularity given in [2] to Banach spaces. We also notice that at least one implication in these characterizations remains valid for the limiting subdifferential without assuming convexity o…
In this article, we investigate nonlinear metric subregularity properties of set-valued mappings between general metric or Banach spaces. We demonstrate that these properties can be treated in the framework of the theory of (linear) error bounds for extended real-valued functions of two variables developed in A. Y. Kruger, Error bounds and metric subregularity, Optimization 64, 1 (2015) 49-79. Several primal and dual space local quantitative and qualitative criteria of nonlinear metric subregularity are formulated. The relationships between the criteria are established and illustrated.
The Holder setting of the metric subregularity property of set-valued mappings between general metric or Banach/Asplund spaces is investigated in the framework of the theory of error bounds for extended real-valued functions of two variables. A classification scheme for the general Holder metric subregularity criteria is presented. The criteria are formulated in terms of several kinds of primal and subdifferential slopes.
In this paper, we mainly study metric subregularity for a convex constraint system defined by a convex set-valued mapping and a convex constraint subset. The main work is to provide several primal equivalent conditions for metric subregularity by contingent cone and graphical derivative. Further it is proved that these primal equivalent conditions can characterize strong basic constraint qualification of convex constraint system given by Zheng and Ng [SIAM J. Optim., 18(2007), pp. 437-460].
In this paper, a systematic study of the strong metric subregularity property of mappings is carried out by means of a variational tool, called steepest displacement rate. With the aid of this tool, a simple characterization of strong metric subregularity for multifunctions acting in metric spaces is formulated. The resulting criterion is shown to be useful for establishing stability properties of the strong metric subregularity in the presence of perturbations, as well as for deriving various conditions, enabling to detect such a property in the case of nonsmooth mappings. Some of these condi…
In this paper, based on basic constraint qualification (BCQ) and strong BCQ for convex generalized equation, we are inspired to further discuss constraint qualifications of BCQ and strong BCQ for nonconvex generalized equation and then establish their various characterizations. As applications, we use these constraint qualifications to study metric subregularity of nonconvex generalized equation and provide necessary and/or sufficient conditions in terms of constraint qualifications considered herein to ensure nonconvex generalized equation having metric subregularity.
Although the property of strong metric subregularity of set-valued mappings has been present in the literature under various names and with various definitions for more than two decades, it has attracted much less attention than its older "siblings", the metric regularity and the strong metric regularity. The purpose of this paper is to show that the strong metric subregularity shares the main features of these two most popular regularity properties and is not less instrumental in applications. We show that the strong metric subregularity of a mapping F acting between metric spaces is stable u…
This paper mainly studies the quadratic growth and the strong metric subregularity of the subdifferential of a function that can be represented as the sum of a function twice differentiable in the extended sense and a subdifferentially continuous, prox-regular, twice epi-differentiable function. For such a function, which is not necessarily prox-regular, it is shown that the quadratic growth, the strong metric subregularity of the subdifferential at a local minimizer, and the positive definiteness of the subgradient graphical derivative at a stationary point are equivalent. In addition, other …
This paper is devoted to the study of metric subregularity and strong subregularity of any positive order $q$ for set-valued mappings in finite and infinite dimensions. While these notions have been studied and applied earlier for $q=1$ and---to a much lesser extent---for $q\in(0,1)$, no results are available for the case $q>1$. We derive characterizations of these notions for subgradient mappings, develop their sensitivity analysis under small perturbations, and provide applications to the convergence rate of Newton-type methods for solving generalized equations.
In this paper we study sufficient conditions for metric subregularity of a set-valued map which is the sum of a single-valued continuous map and a locally closed subset. First we derive a sufficient condition for metric subregularity which is weaker than the so-called first-order sufficient condition for metric subregularity (FOSCMS) by adding an extra sequential condition. Then we introduce a directional version of the quasi-normality and the pseudo-normality which is stronger than the new {weak} sufficient condition for metric subregularity but is weaker than the classical quasi-normality an…
There is a basic paradigm, called here the radius of well-posedness, which quantifies the "distance" from a given well-posed problem to the set of ill-posed problems of the same kind. In variational analysis, well-posedness is often understood as a regularity property, which is usually employed to measure the effect of perturbations and approximations of a problem on its solutions. In this paper we focus on evaluating the radius of the property of metric subregularity which, in contrast to its siblings, metric regularity, strong regularity and strong subregularity, exhibits a more complicated …
In this paper we intend to give some calculus rules for tangent sets in the sense of Bouligand and Ursescu, as well as for corresponding derivatives of set-valued maps. Both first and second order objects are envisaged and the assumptions we impose in order to get the calculus are in terms of metric subregularity of the assembly of the initial data. This approach is different from those used in alternative recent papers in literature and allows us to avoid compactness conditions. A special attention is paid for the case of perturbation set-valued maps which appear naturally in optimization pro…
This paper is devoted to the study of the metric subregularity constraint qualification (MSCQ) for general optimization problems, with the emphasis on the nonconvex setting. We elaborate on notions of directional pseudo- and quasi-normality, recently introduced by Bai et al. (SIAM J. Opt., 2019), which combine the standard approach via pseudo- and quasi-normality with modern tools of directional variational analysis. We focus on applications to disjunctive programs, where (directional) pseudo-normality is characterized via an extremal condition. This, in turn, yields efficient tools to verify …
We describe and analyse Levenberg-Marquardt methods for solving systems of nonlinear equations. More specifically, we propose an adaptive formula for the Levenberg-Marquardt parameter and analyse the local convergence of the method under Hölder metric subregularity of the function defining the equation and Hölder continuity of its gradient mapping. Further, we analyse the local convergence of the method under the additional assumption that the Łojasiewicz gradient inequality holds. We finally report encouraging numerical results confirming the theoretical findings for the problem of computing …
Primal-Dual Error Bounds and KKT Metric Subregularity in Convex Composite Optimization
School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学院）；Beijing Key Laboratory on MCAACI/Key Laboratory of Mathematical Theory and Computation in Information Security, Beijing Institute of Technology（北京理工大学计算机辅助设计与图形学北京市重点实验室/信息安全数学理论与计算北京市重点实验室）
Regularisation theory in Banach spaces, and non--norm-squared regularisation even in finite dimensions, generally relies upon Bregman divergences to replace norm convergence. This is comparable to the extension of first-order optimisation methods to Banach spaces. Bregman divergences can, however, be somewhat suboptimal in terms of descriptiveness. Using the concept of (strong) metric subregularity, previously used to prove the fast local convergence of optimisation methods, we show norm convergence in Banach spaces and for non--norm-squared regularisation. For problems such as total variation…
For the inclusion problem involving two maximal monotone operators, under the metric subregularity of the composite operator, we derive the linear convergence of the generalized proximal point algorithm and several splitting algorithms, which include the over-relaxed forward-backward splitting algorithm, the generalized Douglas-Rachford splitting algorithm and Davis' three-operator splitting algorithm. To the best of our knowledge, this linear convergence condition is weaker than the existing ones that almost all require the strong monotonicity of the composite operator. Withal, we give some s…
In this work, we develop a level-set subdifferential error bound condition aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. It is proved that the aforementioned condition guarantees linear convergence of VBPG, and is weaker than Kurdyka-Lojasiewicz property, weak metric subregularity and Bregman proximal error bound. Along the way, we are able to derive a number of verifiable conditions for level-set subdifferential error bounds to hold, and necessary conditions and sufficient cond…
In this paper, we consider the $\ell_0$ minimization problem whose objective function is the sum of $\ell_0$-norm and convex differentiable function. A variable metric type method which combines the PIHT method and the skill in quasi-newton method, named variable metric extrapolation proximal iterative hard thresholding (VMEPIHT) method, is proposed. Then we analyze its convergence, linear convergence rate and superlinear convergence rate under appropriate assumptions. Finally, we conduct numerical experiments on compressive sensing problem and CT image reconstruction problem to confirm VMPIHT…
We present necessary conditions for monotonicity, in one form or another, of fixed point iterations of mappings that violate the usual nonexpansive property. We show that most reasonable notions of linear-type monotonicity of fixed point sequences imply {\em metric subregularity}. This is specialized to the alternating projections iteration where the metric subregularity property takes on a distinct geometric characterization of sets at points of intersection called {\em subtransversality}. Our more general results for fixed point iterations are specialized to establish the necessity of subtra…
Hebei University of Technology（河北工业大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
Variable metric proximal gradient methods with different metric selections have been widely used in composite optimization. Combining the Barzilai-Borwein (BB) method with a diagonal selection strategy for the metric, the diagonal BB stepsize can keep low per-step computation cost as the scalar BB stepsize and better capture the local geometry of the problem. In this paper, we propose a variable metric mini-batch proximal stochastic recursive gradient algorithm VM-mSRGBB, which updates the metric using a new diagonal BB stepsize. The linear convergence of VM-mSRGBB is established for strongly …
A Stochastic Block-coordinate Proximal Newton Method for Nonconvex Composite Minimization
School of Mathematical Sciences, Jiangsu University（江苏大学数学科学学院）；Center of Excellence for Generative AI, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学生成式人工智能卓越中心）
We propose two numerical algorithms in the fully nonconvex setting for the minimization of the sum of a smooth function and the composition of a nonsmooth function with a linear operator. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its linearized variant, respectively. The proximal terms are introduced via variable metrics, a fact which allows us to derive new proximal splitting algorithms for nonconvex structured optimization problems, as particular instances of the general schemes. Under mild conditions on the sequence of…
We propose a novel algorithmic framework of Variable Metric Over-Relaxed Hybrid Proximal Extra-gradient (VMOR-HPE) method with a global convergence guarantee for the maximal monotone operator inclusion problem. Its iteration complexities and local linear convergence rate are provided, which theoretically demonstrate that a large over-relaxed step-size contributes to accelerating the proposed VMOR-HPE as a byproduct. Specifically, we find that a large class of primal and primal-dual operator splitting algorithms are all special cases of VMOR-HPE. Hence, the proposed framework offers a new insig…
This paper studies a proximal alternating direction method of multipliers (ADMM) with variable metric indefinite proximal terms for linearly constrained convex optimization problems. The proximal ADMM plays an important role in many application areas, since the subproblems of the method are easy to solve. Recently, it is reported that the proximal ADMM with a certain fixed indefinite proximal term is faster than that with a positive semidefinite term, and still has the global convergence property. On the other hand, Gu and Yamashita studied a variable metric semidefinite proximal ADMM whose pr…
In this paper we introduce the concept of modulus of regularity as a tool to analyze the speed of convergence, including the finite termination, for classes of Fejér monotone sequences which appear in fixed point theory, monotone operator theory, and convex optimization. This concept allows for a unified approach to several notions such as weak sharp minima, error bounds, metric subregularity, Hölder regularity, etc., as well as to obtain rates of convergence for Picard iterates, the Mann algorithm, the proximal point algorithm and the cyclic algorithm. As a byproduct we obtain a quantitative …
In this paper the degenerate preconditioned proximal point algorithm will be combined with the idea of varying preconditioners leading to the degenerate variable metric proximal point algorithm. The weak convergence of the resulting iteration will be proven. From the perspective of the degenerate variable metric proximal point algorithm, a version of the primal-dual Douglas-Rachford method with varying preconditioners will be derived and a proof of its weak convergence which is based on the previous results for the proximal point algorithm, is provided, too. After that, we derive a heuristic o…
The proximal point algorithm, which is a well-known tool for finding minima of convex functions, is generalized from the classical Hilbert space framework into a nonlinear setting, namely, geodesic metric spaces of nonpositive curvature. We prove that the sequence generated by the proximal point algorithm weakly converges to a minimizer, and also discuss a related question: convergence of the gradient flow.
In this paper, we obtain global $\mathcal{O} (1/ \sqrt{k})$ pointwise and $\mathcal{O} (1/ {k})$ ergodic convergence rates for a variable metric proximal alternating direction method of multipliers(VM-PADMM) for solving linearly constrained convex optimization problems. The VM-PADMM can be seen as a class of ADMM variants, allowing the use of degenerate metrics (defined by noninvertible linear operators). We first propose and study nonasymptotic convergence rates of a variable metric hybrid proximal extragradient (VM-HPE) framework for solving monotone inclusions. Then, the above-mentioned con…
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
We introduce a framework for quasi-Newton forward--backward splitting algorithms (proximal quasi-Newton methods) with a metric induced by diagonal $\pm$ rank-$r$ symmetric positive definite matrices. This special type of metric allows for a highly efficient evaluation of the proximal mapping. The key to this efficiency is a general proximal calculus in the new metric. By using duality, formulas are derived that relate the proximal mapping in a rank-$r$ modified metric to the original metric. We also describe efficient implementations of the proximity calculation for a large class of functions;…
This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is a new parameterization of the optimality condition which allows us to develop a class of homotopy proximal variable-metric methods. We show that under appropriate assumptions such as strong convexity-type and smoothness, or self-concordance, our new schemes can achieve finite global iteration-complexity bounds. Our second idea is a primal-dual-primal framework for proximal-Newton methods which can lead to some useful computational …
A Decomposed Bilevel Search for Variable-Metric Proximal Gradient Methods
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and University of Chinese Academy of Sciences（中国科学院数学与系统科学研究院、中国科学院大学）；State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
In this paper, we consider a composite difference-of-convex (DC) program, whose objective function is the sum of a smooth convex function with Lipschitz continuous gradient, a proper closed and convex function, and a continuous concave function. This problem has many applications in machine learning and data science. The proximal DCA (pDCA), a special case of the classical DCA, as well as two Nesterov-type extrapolated DCA -- ADCA (Phan et al. IJCAI:1369--1375, 2018) and pDCAe (Wen et al. Comput Optim Appl 69:297--324, 2018) -- can solve this problem. The algorithmic step-sizes of pDCA, pDCAe,…
We propose an inexact variable-metric proximal point algorithm to accelerate gradient-based optimization algorithms. The proposed scheme, called QNing can be notably applied to incremental first-order methods such as the stochastic variance-reduced gradient descent algorithm (SVRG) and other randomized incremental optimization algorithms. QNing is also compatible with composite objectives, meaning that it has the ability to provide exactly sparse solutions when the objective involves a sparsity-inducing regularization. When combined with limited-memory BFGS rules, QNing is particularly effecti…
Stanford University（斯坦福大学）；LG Sillicon Valley Lab（LG硅谷实验室）
Variable metric proximal gradient (VM-PG) is a widely used class of convex optimization method. Lately, there has been a lot of research on the theoretical guarantees of VM-PG with different metric selections. However, most such metric selections are dependent on (an expensive) Hessian, or limited to scalar stepsizes like the Barzilai-Borwein (BB) stepsize with lots of safeguarding. Instead, in this paper we propose an adaptive metric selection strategy called the diagonal Barzilai-Borwein (BB) stepsize. The proposed diagonal selection better captures the local geometry of the problem while ke…
