We consider minimizing the composite function that consists of a strongly convex function and a convex function. The fast dual proximal gradient (FDPG) method decreases the dual function with a rate $O(1/k^2)$, leading to a rate $O(1/k)$ for decreasing the primal function. We propose a generalized FDPG method that guarantees an $O(1/k^{1.5})$ rate for the dual proximal gradient norm decrease. By relating this to the primal function decrease, the proposed approach decreases the primal function with the improved $O(1/k^{1.5})$ rate.
The proximal gradient descent method, well-known for composite optimization, can be completely described by the concept of proximal gradient mapping. In this paper, we highlight our previous two discoveries of proximal gradient mapping--norm monotonicity and refined descent, with which we are able to extend the recently proposed potential function-based framework from gradient descent to proximal gradient descent.
In this note, we provide convergence results for the proximal point algorithm and a splitting variant thereof in the setting of CAT$(κ)$ spaces with $κ> 0$ using a recent definition for the resolvent of a convex, lower semi-continuous function due to Kimura and Kohsaka (J. Fixed Point Theory Appl. 18 (2016), 93-115).
In their 2016 article `Meetings with Lambert $\mathcal{W}$ and Other Special Functions in Optimization and Analysis', Borwein and Lindstrom considered the minimization of an entropy functional utilizing the weighted average of the negative Boltzmann--Shannon entropy $x\mapsto x\log x -x$ and the energy $x \mapsto x^2/2$; the solution employed the Fenchel-Moreau conjugate of the sum. However, in place of a traditional arithmetic average of two convex functions, it is also, perhaps even more, natural to use their proximal average. We explain the advantages and illustrate them by computing the an…
In this paper, we present a semi-proximal alternating direction method of multipliers (ADMM) for solving $3$-block separable convex minimization problems with the second block in the objective being a strongly convex function and one coupled linear equation constraint. By choosing the semi-proximal terms properly, we establish the global convergence of the proposed semi-proximal ADMM for the step-length $τ\in (0, (1+\sqrt{5})/2)$ and the penalty parameter $σ\in (0, +\infty)$. In particular, if $σ>0$ is smaller than a certain threshold and the first and third linear operators in the linear equa…
In this paper, we consider a non-convex problem which is the sum of $\ell_0$-norm and a convex smooth function under box constraint. We propose one proximal iterative hard thresholding type method with extrapolation step used for acceleration and establish its global convergence results. In detail, the sequence generated by the proposed method globally converges to a local minimizer of the objective function. Finally, we conduct numerical experiments to show the proposed method's effectiveness on comparison with some other efficient methods.
In this paper we propose a scalarization proximal point method to solve multiobjective unconstrained minimization problems with locally Lipschitz and quasiconvex vector functions. We prove, under natural assumptions, that the sequence generated by the method is well defined and converges globally to a Pareto-Clarke critical point. Our method may be seen as an extension, for the non convex case, of the inexact proximal method for multiobjective convex minimization problems studied by Bonnel et al. (SIAM Journal on Optimization 15, 4, 953-970, 2005).
In this paper we propose a proximal algorithm for minimizing an objective function of two block variables consisting of three terms: 1) a smooth function, 2) a nonsmooth function which is a composition between a strictly increasing, concave, differentiable function and a convex nonsmooth function, and 3) a smooth function which couples the two block variables. We propose a variable metric composite proximal alternating linearized minimization (CPALM) to solve this class of problems. Building on the powerful Kurdyka-Łojasiewicz property, we derive the convergence analysis and establish that eac…
In this paper we study nonconvex and nonsmooth optimization problems with semi-algebraic data, where the variables vector is split into several blocks of variables. The problem consists of one smooth function of the entire variables vector and the sum of nonsmooth functions for each block separately. We analyze an inertial version of the Proximal Alternating Linearized Minimization (PALM) algorithm and prove its global convergence to a critical point of the objective function at hand. We illustrate our theoretical findings by presenting numerical experiments on blind image deconvolution, on sp…
Saddle point with a given Morse index on a potential energy surface is an important object related to energy landscape in physics and chemistry. Efficient numerical methods based on iterative minimization formulation have been proposed in the forms of the sequence of minimization subproblems or the continuous dynamics. We here present a differential game interpretation of this formulation and theoretically investigate the Nash equilibrium of the proposed game and the original saddle point on potential energy surface. To define this differential game, a new proximal function growing faster than…
We consider the estimation of the state transition matrix in vector autoregressive models, when time sequence data is limited but nonsequence steady-state data is abundant. To leverage both sources of data, we formulate the least squares minimization problem regularized by a Lyapunov penalty. We impose cardinality or rank constraints to reduce the complexity of the autoregressive model. We solve the resulting nonconvex, nonsmooth problem by using the proximal alternating linearization method (PALM). We show that PALM is globally convergent to a critical point and that the estimation error mono…
We consider the problem of minimizing the sum of two convex functions: one is smooth and given by a gradient oracle, and the other is separable over blocks of coordinates and has a simple known structure over each block. We develop an accelerated randomized proximal coordinate gradient (APCG) method for minimizing such convex composite functions. For strongly convex functions, our method achieves faster linear convergence rates than existing randomized proximal coordinate gradient methods. Without strong convexity, our method enjoys accelerated sublinear convergence rates. We show how to apply…
We propose a proximal algorithm for minimizing objective functions consisting of three summands: the composition of a nonsmooth function with a linear operator, another nonsmooth function, each of the nonsmooth summands depending on an independent block variable, and a smooth function which couples the two block variables. The algorithm is a full splitting method, which means that the nonsmooth functions are processed via their proximal operators, the smooth function via gradient steps, and the linear operator via matrix times vector multiplication. We provide sufficient conditions for the bou…
In this paper we propose a linear scalarization proximal point algorithm for solving arbitrary lower semicontinuous quasiconvex multiobjective minimization problems. Under some natural assumptions and using the condition that the proximal parameters are bounded we prove the convergence of the sequence generated by the algorithm and when the objective functions are continuous, we prove the convergence to a generalized critical point. Furthermore, if each iteration minimize the proximal regularized function and the proximal parameters converges to zero we prove the convergence to a weak Pareto s…
In this paper, we consider an accelerated method for solving nonconvex and nonsmooth minimization problems. We propose a Bregman Proximal Gradient algorithm with extrapolation(BPGe). This algorithm extends and accelerates the Bregman Proximal Gradient algorithm (BPG), which circumvents the restrictive global Lipschitz gradient continuity assumption needed in Proximal Gradient algorithms (PG). The BPGe algorithm has higher generality than the recently introduced Proximal Gradient algorithm with extrapolation(PGe), and besides, due to the extrapolation step, BPGe converges faster than BPG algori…
We provide comparison principles for convex functions through its proximal mappings. Consequently, we prove that the norm of the proximal operator determines a convex the function up to a constant. A new characterization of Lipschitzianity in terms of the proximal operator is given.
We consider the exact continuous relaxation model of matrix rank minimization problem proposed by Yu and Zhang (Comput.Optim.Appl. 1-20, 2022). Motivated by the inertial techinique, we propose a general inertial smoothing proximal gradient algorithm(GIMSPG) for this kind of problems. It is shown that the singular values of any accumulation point have a common support set and the nonzero singular values have a unified lower bound. Besides, the zero singular values of the accumulation point can be achieved within finite iterations. Moreover, we prove that any accumulation point of the sequence g…
We study the convergence rate of the proximal-gradient homotopy algorithm applied to norm-regularized linear least squares problems, for a general class of norms. The homotopy algorithm reduces the regularization parameter in a series of steps, and uses a proximal-gradient algorithm to solve the problem at each step. Proximal-gradient algorithm has a linear rate of convergence given that the objective function is strongly convex, and the gradient of the smooth component of the objective function is Lipschitz continuous. In many applications, the objective function in this type of problem is no…
We study the worst-case convergence rates of the proximal gradient method for minimizing the sum of a smooth strongly convex function and a non-smooth convex function whose proximal operator is available. We establish the exact worst-case convergence rates of the proximal gradient method in this setting for any step size and for different standard performance measures: objective function accuracy, distance to optimality and residual gradient norm. The proof methodology relies on recent developments in performance estimation of first-order methods based on semidefinite programming. In the case …
In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proxima…
We give a novel proof of the $O(1/k)$ and $O(1/k^2)$ convergence rates of the proximal gradient and accelerated proximal gradient methods for composite convex minimization. The crux of the new proof is an upper bound constructed via the convex conjugate of the objective function.
Consider the minimization of a nonconvex differentiable function over a polyhedron. A popular primal-dual first-order method for this problem is to perform a gradient projection iteration for the augmented Lagrangian function and then update the dual multiplier vector using the constraint residual. However, numerical examples show that this approach can exhibit "oscillation" and may not converge. In this paper, we propose a proximal alternating direction method of multipliers for the multi-block version of this problem. A distinctive feature of this method is the introduction of a "smoothed" (…
We provide a unifying framework for distributed convex optimization over time-varying networks, in the presence of constraints and uncertainty, features that are typically treated separately in the literature. We adopt a proximal minimization perspective and show that this set-up allows us to bypass the difficulties of existing algorithms while simplifying the underlying mathematical analysis. We develop an iterative algorithm and show convergence of the resulting scheme to some optimizer of the centralized problem. To deal with the case where the agents' constraint sets are affected by a poss…
In this paper, we study the convergence of an interior subgradient and proximal methods for a DC (difference of convex functions) constrained minimization problem.
This paper is concerned with a class of zero-norm regularized piecewise linear-quadratic (PLQ) composite minimization problems, which covers the zero-norm regularized $\ell_1$-loss minimization problem as a special case. For this class of nonconvex nonsmooth problems, we show that its equivalent MPEC reformulation is partially calm on the set of global optima and make use of this property to derive a family of equivalent DC surrogates. Then, we propose a proximal majorization-minimization (MM) method, a convex relaxation approach not in the DC algorithm framework, for solving one of the DC sur…
We propose a new first-order optimisation algorithm to solve high-dimensional non-smooth composite minimisation problems. Typical examples of such problems have an objective that decomposes into a non-smooth empirical risk part and a non-smooth regularisation penalty. The proposed algorithm, called Semi-Proximal Mirror-Prox, leverages the Fenchel-type representation of one part of the objective while handling the other part of the objective via linear minimization over the domain. The algorithm stands in contrast with more classical proximal gradient algorithms with smoothing, which require th…
We provide a proximal average with repect to a $1$-coercive Legendre function. In the sense of Bregman distance, the Bregman envelope of the proximal average is a convex combination of Bregman envelopes of individual functions. The Bregman proximal mapping of the average is a convex combination of convexified proximal mappings of individual functions. Techniques from variational analysis provide the keys for the Bregman proximal average.
We present Epsilon, a system for general convex programming using fast linear and proximal operators. As with existing convex programming frameworks, users specify convex optimization problems using a natural grammar for mathematical expressions, composing functions in a way that is guaranteed to be convex by the rules of disciplined convex programming. Given such an input, the Epsilon compiler transforms the optimization problem into a mathematically equivalent form consisting only of functions with efficient proximal operators---an intermediate representation we refer to as prox-affine form.…
We propose new proximal bundle algorithms for minimizing a nonsmooth convex function. These algorithms are derived from the application of Nesterov fast gradient methods for smooth convex minimization to the so-called Moreau-Yosida regularization $F_μ$ of $f$ w.r.t. some $μ>0$. Since the exact values and gradients of $F_μ$ are difficult to evaluate, we use approximate proximal points thanks to a bundle strategy to get implementable algorithms. One of these algorithms appears as an implementable version of a special case of inertial proximal algorithm. We give their complexity estimates in term…
This paper studies a proximal alternating direction method of multipliers (ADMM) with variable metric indefinite proximal terms for linearly constrained convex optimization problems. The proximal ADMM plays an important role in many application areas, since the subproblems of the method are easy to solve. Recently, it is reported that the proximal ADMM with a certain fixed indefinite proximal term is faster than that with a positive semidefinite term, and still has the global convergence property. On the other hand, Gu and Yamashita studied a variable metric semidefinite proximal ADMM whose pr…
In the lines of our approach in \cite{Ouorou2019}, where we exploit Nesterov fast gradient concept \cite{Nesterov1983} to the Moreau-Yosida regularization of a convex function, we devise new proximal algorithms for nonsmooth convex optimization. These algorithms need no bundling mechanism to update the stability center while preserving the complexity estimates established in \cite{Ouorou2019}. We report some preliminary computational results on some academic test problem to give a first estimate of their performance in relation with the classical proximal bundle algorithm.
We investigate the asymptotic properties of the trajectories generated by a second-order dynamical system of proximal-gradient type stated in connection with the minimization of the sum of a nonsmooth convex and a (possibly nonconvex) smooth function. The convergence of the generated trajectory to a critical point of the objective is ensured provided a regularization of the objective function satisfies the Kurdyka-Łojasiewicz property. We also provide convergence rates for the trajectory formulated in terms of the Łojasiewicz exponent.
This paper suggests two novel ideas to develop new proximal variable-metric methods for solving a class of composite convex optimization problems. The first idea is a new parameterization of the optimality condition which allows us to develop a class of homotopy proximal variable-metric methods. We show that under appropriate assumptions such as strong convexity-type and smoothness, or self-concordance, our new schemes can achieve finite global iteration-complexity bounds. Our second idea is a primal-dual-primal framework for proximal-Newton methods which can lead to some useful computational …
This paper extends the algorithm schemes proposed in \cite{Nesterov2007a} and \cite{Nesterov2007b} to the minimization of the sum of a composite objective function and a convex function. Two proximal point-type schemes are provided and their global convergence is investigated. The worst case complexity bound is also estimated under certain Lipschitz conditions and nondegeneratedness. The algorithm is then accelerated to get a faster convergence rate for the strongly convex case.
Structured convex optimization problems typically involve a mix of smooth and nonsmooth functions. The common practice is to activate the smooth functions via their gradient and the nonsmooth ones via their proximity operator. We show that, although intuitively natural, this approach is not necessarily the most efficient numerically and that, in particular, activating all the functions proximally may be advantageous. To make this viewpoint viable computationally, we derive a number of new examples of proximity operators of smooth convex functions arising in applications. A novel variational mo…
This article develops a primal dual formulation for a primal proximal approach suitable for a large class of non-convex models in the calculus of variations. The results are established through standard tools of functional analysis, convex analysis and duality theory and are applied to a Ginzburg-Landau type model. Finally, in the last two sections, we present concerning optimal conditions and another related duality principle for the model in question.
We present two approximate versions of the proximal subgradient method for minimizing the sum of two convex functions (not necessarily differentiable). The algorithms involve, at each iteration, inexact evaluations of the proximal operator and approximate subgradients of the functions (namely: the $ε$-subgradients). The methods use different error criteria for approximating the proximal operators. We provide an analysis of the convergence and rate of convergence properties of these methods, considering various stepsize rules, including both, diminishing and constant stepsizes. For the case whe…
Total variation (TV) is a widely used regularizer for stabilizing the solution of ill-posed inverse problems. In this paper, we propose a novel proximal-gradient algorithm for minimizing TV regularized least-squares cost functional. Our method replaces the standard proximal step of TV by a simpler alternative that computes several independent proximals. We prove that the proposed parallel proximal method converges to the TV solution, while requiring no sub-iterations. The results in this paper could enhance the applicability of TV for solving very large scale imaging inverse problems.
This paper focuses on stochastic proximal gradient methods for optimizing a smooth non-convex loss function with a non-smooth non-convex regularizer and convex constraints. To the best of our knowledge we present the first non-asymptotic convergence results for this class of problem. We present two simple stochastic proximal gradient algorithms, for general stochastic and finite-sum optimization problems, which have the same or superior convergence complexities compared to the current best results for the unconstrained problem setting. In a numerical experiment we compare our algorithms with t…
In this paper, we develop a parameterized proximal point algorithm (P-PPA) for solving a class of separable convex programming problems subject to linear and convex constraints. The proposed algorithm is provable to be globally convergent with a worst-case O(1/t) convergence rate, wheret denotes the iteration number. By properly choosing the algorithm parameters, numerical experiments on solving a sparse optimization problem arising from statistical learning show that our P-PPA could perform significantly better than other state-of-the-art methods, such as the alternating direction method of m…
We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: $L(x,y)=f(x)+Q(x,y)+g(y)$, where $f:\R^n\rightarrow\R\cup{+\infty}$ and $g:\R^m\rightarrow\R\cup{+\infty}$ are proper lower semicontinuous functions, and $Q:\R^n\times\R^m\rightarrow \R$ is a smooth $C^1$ function which couples the variables $x$ and $y$. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize $L$. We work in a nonconvex setting, just assuming that the function $L$ satisfies the Kurdyka-Łojasiewicz in…
In this paper we present a variant of the proximal forward-backward splitting iteration for solving nonsmooth optimization problems in Hilbert spaces, when the objective function is the sum of two nondifferentiable convex functions. The proposed iteration, which will be called Proximal Subgradient Splitting Method, extends the classical subgradient iteration for important classes of problems, exploiting the additive structure of the objective function. The weak convergence of the generated sequence was established using different stepsizes and under suitable assumptions. Moreover, we analyze t…
We propose an algorithmic framework for convex minimization problems of a composite function with two terms: a self-concordant function and a possibly nonsmooth regularization term. Our method is a new proximal Newton algorithm that features a local quadratic convergence rate. As a specific instance of our framework, we consider the sparse inverse covariance matrix estimation in graph learning problems. Via a careful dual formulation and a novel analytic step-size selection procedure, our approach for graph learning avoids Cholesky decompositions and matrix inversions in its iteration making i…
This note studies the distributed non-convex optimization problem with non-smooth regularization, which has wide applications in decentralized learning, estimation and control. The objective function is the sum of different local objective functions, which consist of differentiable (possibly non-convex) cost functions and non-smooth convex functions. This paper presents a distributed proximal gradient algorithm for the non-smooth non-convex optimization problem over time-varying multi-agent networks. Each agent updates local variable estimate by the multi-step consensus operator and the proxim…
We show that Dykstra's splitting for projecting onto the intersection of convex sets can be extended to minimize the sum of convex functions and a regularizing quadratic. We give conditions for which convergence to the primal minimizer holds so that more than one convex function can be minimized at a time, the convex functions are not necessarily sampled in a cyclic manner, and the SHQP strategy for problems involving the intersection of more than one convex set can be applied. When the sum does not involve the regularizing quadratic, we discuss an approximate proximal point method combined wi…
The breakthrough ideas in the modern proximal splitting methodologies allow us to express the set of all minimizers of a superposition of multiple nonsmooth convex functions as the fixed point set of computable nonexpansive operators. In this paper, we present practical algorithmic strategies for the hierarchical convex optimization problems which require further strategic selection of a most desirable vector from the solution set of the standard convex optimization. The proposed algorithms are established by applying the hybrid steepest descent method to special nonexpansive operators designe…
