In this paper, we consider a class of generalized difference-of-convex functions (DC) programming, whose objective is the difference of two convex (not necessarily smooth) functions plus a decomposable (possibly nonconvex) function with Lipschitz gradient. By employing the Fenchel-Young inequality and Moreau decomposition theorem, we introduce an inertial Bregman proximal DC algorithm to solve the problem under consideration. Our algorithmic framework is able to fully exploit the decomposable structure of the generalized DC programming such that each subproblem of the algorithm is enough easy in many cases. Theoretically, we show that the sequence generated by the proposed algorithm globally converges to a critical point under the Kurdyka-Łojasiewicz condition. A series of numerical results demonstrate that our algorithm runs efficiently on matrix and tensor completion problems.
We provide a proximal average with repect to a $1$-coercive Legendre function. In the sense of Bregman distance, the Bregman envelope of the proximal average is a convex combination of Bregman envelopes of individual functions. The Bregman proximal mapping of the average is a convex combination of convexified proximal mappings of individual functions. Techniques from variational analysis provide the keys for the Bregman proximal average.
In this paper, we provide a simple convergence analysis of proximal gradient algorithm with Bregman distance, which provides a tighter bound than existing result. In particular, for the problem of minimizing a class of convex objective functions, we show that proximal gradient algorithm with Bregman distance can be viewed as proximal point algorithm that incorporates another Bregman distance. Consequently, the convergence result of the proximal gradient algorithm with Bregman distance follows directly from that of the proximal point algorithm with Bregman distance, and this leads to a simpler …
We systematically study the local single-valuedness of the Bregman proximal mapping and local smoothness of the Bregman--Moreau envelope of a nonconvex function under relative prox-regularity - an extension of prox-regularity - which was originally introduced by Poliquin and Rockafellar. As Bregman distances are asymmetric in general, in accordance with Bauschke et al., it is natural to consider two variants of the Bregman proximal mapping, which, depending on the order of the arguments, are called left and right Bregman proximal mapping. We consider the left Bregman proximal mapping first. Th…
In this paper we study the problems of minimizing the sum of two nonconvex functions: one is differentiable and satisfies smooth adaptable property. The smooth adaptable property, also named relatively smooth condition, is weaker than the globally gradient Lipschitz continuity. We analyze an inertial version of the Bregman Proximal Gradient (BPG) algorithm and prove its stationary convergence. Besides, we prove a sublinear convergence of the inertial algorithm. Moreover, if the objective function satisfies Kurdyka--Łojasiewicz (KL) property, its global convergence to a critical point of the ob…
In this paper we present the bilevel equilibrium problem under conditions of pseudomonotonicity. Using Bregman distances on Hadamard manifolds we propose a framework for to analyse the convergence of a proximal point algorithm to solve this bilevel equilibrium problem. As an application, we consider the problem of "how routines form and change" which is crucial for the dynamics of organizations in Economics and Management Sciences.
In this work, we develop a level-set subdifferential error bound condition aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. It is proved that the aforementioned condition guarantees linear convergence of VBPG, and is weaker than Kurdyka-Lojasiewicz property, weak metric subregularity and Bregman proximal error bound. Along the way, we are able to derive a number of verifiable conditions for level-set subdifferential error bounds to hold, and necessary conditions and sufficient cond…
We develop a new variational approach on level sets aiming towards convergence rate analysis of a variable Bregman proximal gradient (VBPG) method for a broad class of nonsmooth and nonconvex optimization problems. With this new approach, we are able to extend the concepts of Bregman proximal mapping and their corresponding Bregman proximal envelops, Bregman proximal gap function to nonconvex setting. Properties of these mappings and functions are examined. An aim of this work is to provide a solid foundation on which further design and analysis of VBPG for more general nonconvex optimization …
In this paper, we consider an accelerated method for solving nonconvex and nonsmooth minimization problems. We propose a Bregman Proximal Gradient algorithm with extrapolation(BPGe). This algorithm extends and accelerates the Bregman Proximal Gradient algorithm (BPG), which circumvents the restrictive global Lipschitz gradient continuity assumption needed in Proximal Gradient algorithms (PG). The BPGe algorithm has higher generality than the recently introduced Proximal Gradient algorithm with extrapolation(PGe), and besides, due to the extrapolation step, BPGe converges faster than BPG algori…
New Bregman proximal type algorithms for solving DC optimization problems
The Graduate University for Advanced Studies（高级研究大学）；University of São Paulo（圣保罗大学）；Tokyo Institute of Technology（东京工业大学）；São Paulo State Technological College（圣保罗州立技术学院）；The Institute of Statistical Mathematics（统计数学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究院先进智能项目中心）
We introduce a unified algorithmic framework, called proximal-like incremental aggregated gradient (PLIAG) method, for minimizing the sum of a convex function that consists of additive relatively smooth convex components and a proper lower semi-continuous convex regularization function, over an abstract feasible set whose geometry can be captured by using the domain of a Legendre function. The PLIAG method includes many existing algorithms in the literature as special cases such as the proximal gradient method, the Bregman proximal gradient method (also called NoLips algorithm), the incrementa…
We consider the problem of minimizing the sum of two convex functions: one is differentiable and relatively smooth with respect to a reference convex function, and the other can be nondifferentiable but simple to optimize. We investigate a triangle scaling property of the Bregman distance generated by the reference convex function and present accelerated Bregman proximal gradient (ABPG) methods that attain an $O(k^{-γ})$ convergence rate, where $γ\in(0,2]$ is the triangle scaling exponent (TSE) of the Bregman distance. For the Euclidean distance, we have $γ=2$ and recover the convergence rate …
Composite and Stochastic Smoothed Decentralized Bregman Proximal-Gradient Methods
The Chinese University of Hong Kong（香港中文大学）；Xiangtan University（湘潭大学）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）；Guangzhou Nanfang College（广州南方学院）
We propose a Bregman inertial forward-reflected-backward (BiFRB) method for nonconvex composite problems. Our analysis relies on a novel approach that imposes general conditions on implicit merit function parameters, which yields a stepsize condition that is independent of inertial parameters. In turn, a question of Malitsky and Tam regarding whether FRB can be equipped with a Nesterov-type acceleration is resolved. Assuming the generalized concave Kurdyka-Łojasiewicz property of a quadratic regularization of the objective, we obtain sequential convergence of BiFRB, as well as convergence rate…
We study a general convex optimization problem, which covers various classic problems in different areas and particularly includes many optimal transport related problems arising in recent years. To solve this problem, we revisit the classic Bregman proximal point algorithm (BPPA) and introduce a new inexact stopping condition for solving the subproblems, which can circumvent the underlying feasibility difficulty often appearing in existing inexact conditions when the problem has a complex feasible set. Our inexact condition also covers several existing inexact conditions as special cases and …
We study the Bregman Augmented Lagrangian method (BALM) for solving convex problems with linear constraints. For classical Augmented Lagrangian method, the convergence rate and its relation with the proximal point method is well-understood. However, the convergence rate for BALM has not yet been thoroughly studied in the literature. In this paper, we analyze the convergence rates of BALM in terms of the primal objective as well as the feasibility violation. We also develop, for the first time, an accelerated Bregman proximal point method, that improves the convergence rate from $O(1/\sum_{k=0}…
Majorization-minimization Bregman proximal gradient algorithms for NMF with the Kullback--Leibler divergence
Graduate School of Information Science and Technology, The University of Tokyo（东京大学大学院信息理工学系研究科）；Department of Statistical Inference and Mathematics, The Institute of Statistical Mathematics（统计数学研究所统计推断与数学系）；The Graduate University for Advanced Studies（日本高级研究院）
Iowa State University（爱荷华州立大学）；IBM Research AI（IBM研究院人工智能部门）；IBM Thomas J. Waston Research Center（IBM托马斯·J·沃森研究中心）
In the applications of signal processing and data analytics, there is a wide class of non-convex problems whose objective function is freed from the common global Lipschitz continuous gradient assumption (e.g., the nonnegative matrix factorization (NMF) problem). Recently, this type of problem with some certain special structures has been solved by Bregman proximal gradient (BPG). This inspires us to propose a new Block-wise two-references Bregman proximal gradient (B2B) method, which adopts two reference functions so that a closed-form solution in the Bregman projection is obtained. Based on …
We propose BIBPA, a block inertial Bregman proximal algorithm for minimizing the sum of a block relatively smooth function (that is, relatively smooth concerning each block) and block separable nonsmooth nonconvex functions. We prove that the sequence generated by BIBPA subsequentially converges to critical points of the objective under standard assumptions, and globally converges when the objective function is additionally assumed to satisfy the Kurdyka-Łojasiewicz (KŁ) property. We also provide the convergence rate when the objective satisfies the Łojasiewicz inequality. We apply BIBPA to th…
In this paper, we introduce an inertial version of the Proximal Incremental Aggregated Gradient method (PIAG) for minimizing the sum of smooth convex component functions and a possibly nonsmooth convex regularization function. Theoretically, we show that the inertial Proximal Incremental Aggregated Gradiend (iPIAG) method enjoys a global linear convergence under a quadratic growth condition, which is strictly weaker than strong convexity, provided that the stepsize is not larger than a constant. Moreover, we present two numerical expreiments which demonstrate that iPIAG outperforms the origina…
Technical University of Munich（慕尼黑工业大学）；Saarland University（萨尔兰大学）
A typical assumption for the analysis of first order optimization methods is the Lipschitz continuity of the gradient of the objective function. However, for many practical applications this assumption is violated, including loss functions in deep learning. To overcome this issue, certain extensions based on generalized proximity measures known as Bregman distances were introduced. This initiated the development of the Bregman proximal gradient (BPG) algorithm and an inertial variant (momentum based) CoCaIn BPG, which however rely on problem dependent Bregman distances. In this paper, we devel…
Blind Deconvolution with Non-smooth Regularization via Bregman Proximal DCAs
The Graduate University for Advanced Studies（advanced studies高级大学）；The Institute of Statistical Mathematics（统计数学研究所）；RIKEN Center for Advanced Intelligence Project（理化学研究院先进智能项目中心）
We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. It turns out that the sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under KL inequality assumption. Further, the rate of…
Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size
University of Utah（犹他大学）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）；Guangzhou Nanfang College（广州南方学院）；The Chinese University of Hong Kong（香港中文大学）
In this paper, we first propose a general inertial proximal point method for the mixed variational inequality (VI) problem. Based on our knowledge, without stronger assumptions, convergence rate result is not known in the literature for inertial type proximal point methods. Under certain conditions, we are able to establish the global convergence and a $o(1/k)$ convergence rate result (under certain measure) of the proposed general inertial proximal point method. We then show that the linearized alternating direction method of multipliers (ADMM) for separable convex optimization with linear co…
The rate of convergence of Bregman proximal methods: Local geometry vs. regularity vs. sharpness
Univ. Grenoble Alpes（格勒诺布尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Grenoble INP（格勒诺布尔理工学院）；Institut de Mathématiques de Toulouse（图卢兹数学研究所）；Université de Toulouse（图卢兹大学）；UPS（图卢兹第三大学）；LIG（格勒诺布尔信息与信号处理实验室）
This paper proposes a two-point inertial proximal point algorithm to find zero of maximal monotone operators in Hilbert spaces. We obtain weak convergence results and non-asymptotic $O(1/n)$ convergence rate of our proposed algorithm in non-ergodic sense. Applications of our results to various well-known convex optimization methods, such as the proximal method of multipliers and the alternating direction method of multipliers are given. Numerical results are given to demonstrate the accelerating behaviors of our method over other related methods in the literature.
We study inertial versions of primal-dual proximal splitting, also known as the Chambolle--Pock method. Our starting point is the preconditioned proximal point formulation of this method. By adding correctors corresponding to the anti-symmetric part of the relevant monotone operator, using a FISTA-style gap unrolling argument, we are able to derive gap estimates instead of merely ergodic gap estimates. Moreover, based on adding a diagonal component to this corrector, we are able to combine strong convexity based acceleration with inertial acceleration. We test our proposed method on image proc…
In this paper, we propose and study the asymptotic convergence and nonasymptotic global convergence rates (iteration-complexity) of an inertial under-relaxed version of the relative-error hybrid proximal extragradient (HPE) method for solving monotone inclusion problems. We analyze the proposed method under more flexible assumptions than existing ones on the extrapolation and relative-error parameters. As applications, we propose and/or study inertial under-relaxed forward-backward and Tseng's modified forward-backward type methods for solving structured monotone inclusions.
