Accelerated proximal gradient methods have recently been developed for solving quasi-static incremental problems of elastoplastic analysis with some different yield criteria. It has been demonstrated through numerical experiments that these methods can outperform conventional optimization-based approaches in computational plasticity. However, in literature these algorithms are described individually for specific yield criteria, and hence there exists no guide for application of the algorithms to other yield criteria. This short paper presents a general form of algorithm design, independent of specific forms of yield criteria, that unifies the existing proximal gradient methods. Clear interpretation is also given to each step of the presented general algorithm so that each update rule is linked to the underlying physical laws in terms of mechanical quantities.
We discuss a topology optimization problem for an elastoplastic medium. The distribution of material in a region is optimized with respect to a given target functional taking into account compliance. The incremental elastoplastic problem serves as state constraint. We prove that the topology optimization problem admits a solution. First-order optimality conditions are obtained by considering a regularized problem and passing to the limit.
In this paper, we introduce an inertial version of the Proximal Incremental Aggregated Gradient method (PIAG) for minimizing the sum of smooth convex component functions and a possibly nonsmooth convex regularization function. Theoretically, we show that the inertial Proximal Incremental Aggregated Gradiend (iPIAG) method enjoys a global linear convergence under a quadratic growth condition, which is strictly weaker than strong convexity, provided that the stepsize is not larger than a constant. Moreover, we present two numerical expreiments which demonstrate that iPIAG outperforms the origina…
Multi-forecast model predictive control (MF-MPC) is a control policy that creates a plan of actions over a horizon for each of a given set of forecasted scenarios or contingencies, with the constraint that the first action in all plans be the same. In this note we show how these coupled plans can be found by solving a sequence of single plans, using an incremental proximal method. We refer to this policy as incremental proximal model predictive control (IP-MPC). We have observed that even when the iterations in IP-MPC are not carried out to convergence, we obtain a policy that achieves much of…
In this paper, we adapt proximal incremental aggregated gradient methods to saddle point problems, which is motivated by decoupling linear transformations in regularized empirical risk minimization models. First, the Primal-Dual Proximal Incremental Aggregated (PD-PIAG) methods with extrapolations were proposed. We proved that the primal-dual gap of the averaged iteration sequence sublinearly converges to 0, and the iteration sequence converges to some saddle point. Under the strong convexity of $f$ and $h^\ast$, we proved that the iteration sequence linearly converges to the saddle point. The…
We introduce a unified algorithmic framework, called proximal-like incremental aggregated gradient (PLIAG) method, for minimizing the sum of a convex function that consists of additive relatively smooth convex components and a proper lower semi-continuous convex regularization function, over an abstract feasible set whose geometry can be captured by using the domain of a Legendre function. The PLIAG method includes many existing algorithms in the literature as special cases such as the proximal gradient method, the Bregman proximal gradient method (also called NoLips algorithm), the incrementa…
We consider the problem of minimizing a finite sum of convex functions subject to the set of minimizers of a convex differentiable function. In order to solve the problem, an algorithm combining the incremental proximal gradient method with smooth penalization technique is proposed. We show the convergence of the generated sequence of iterates to an optimal solution of the optimization problems, provided that a condition expressed via the Fenchel conjugate of the constraint function is fulfilled. Finally, the functionality of the method is illustrated by some numerical experiments addressing i…
In this paper, we study the proximal incremental aggregated gradient(PIAG) algorithm for minimizing the sum of L-smooth nonconvex component functions and a proper closed convex function. By exploiting the L-smooth property and with the help of an error bound condition, we can show that the PIAG method still enjoys some nice linear convergence properties even for nonconvex minimization. To illustrate this, we first demonstrate that the generated sequence globally converges to the stationary point set. Then, there exists a threshold such that the objective function value sequence and the iterate…
In this paper, we propose a probabilistic optimization method, named probabilistic incremental proximal gradient (PIPG) method, by developing a probabilistic interpretation of the incremental proximal gradient algorithm. We explicitly model the update rules of the incremental proximal gradient method and develop a systematic approach to propagate the uncertainty of the solution estimate over iterations. The PIPG algorithm takes the form of Bayesian filtering updates for a state-space model constructed by using the cost function. Our framework makes it possible to utilize well-known exact or ap…
In this work, we highlight a connection between the incremental proximal method and stochastic filters. We begin by showing that the proximal operators coincide, and hence can be realized with, Bayes updates. We give the explicit form of the updates for the linear regression problem and show that there is a one-to-one correspondence between the proximal operator of the least-squares regression and the Bayes update when the prior and the likelihood are Gaussian. We then carry out this observation to a general sequential setting: We consider the incremental proximal method, which is an algorithm…
Despite the rise to fame of incremental variance-reduced methods in recent years, their use in nonsmooth optimization is still limited to few simple cases. This is due to the fact that existing methods require to evaluate the proximity operator for the nonsmooth terms, which can be a costly operation for complex penalties. In this work we introduce two variance-reduced incremental methods based on SAGA and SVRG that can efficiently take into account complex penalties which can be expressed as a sum of proximal terms. This includes penalties such as total variation, group lasso with overlap and…
National University of Defense Technology（国防科技大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
The incremental aggregated gradient algorithm is popular in network optimization and machine learning research. However, the current convergence results require the objective function to be strongly convex. And the existing convergence rates are also limited to linear convergence. Due to the mathematical techniques, the stepsize in the algorithm is restricted by the strongly convex constant, which may make the stepsize be very small (the strongly convex constant may be small). In this paper, we propose a general proximal incremental aggregated gradient algorithm, which contains various existin…
In this paper, we consider the dual formulation of minimizing $\sum_{i\in I}f_i(x_i)+\sum_{j\in J} g_j(\mathcal{A}_jx)$ with the index sets $I$ and $J$ being large. To address the difficulties from the high dimension of the variable $x$ (i.e., $I$ is large) and the large number of component functions $g_j$ (i.e., $J$ is large), we propose a hybrid method called the random dual coordinate incremental aggregated gradient method by blending the random dual block coordinate descent method and the proximal incremental aggregated gradient method. To the best of our knowledge, no research is done to …
We propose an inexact variable-metric proximal point algorithm to accelerate gradient-based optimization algorithms. The proposed scheme, called QNing can be notably applied to incremental first-order methods such as the stochastic variance-reduced gradient descent algorithm (SVRG) and other randomized incremental optimization algorithms. QNing is also compatible with composite objectives, meaning that it has the ability to provide exactly sparse solutions when the objective involves a sparsity-inducing regularization. When combined with limited-memory BFGS rules, QNing is particularly effecti…
On Tackling High-Dimensional Nonconvex Stochastic Optimization via Stochastic First-Order Methods with Non-smooth Proximal Terms and Variance Reduction
The University of Hong Kong（香港大学）；The University of Minnesota Twin Cities（明尼苏达大学双城分校）；University of Florida（佛罗里达大学）
Motivated by applications arising from sensor networks and machine learning, we consider the problem of minimizing a finite sum of nondifferentiable convex functions where each component function is associated with an agent and a hard-to-project constraint set. Among well-known avenues to address finite sum problems is the class of incremental gradient (IG) methods where a single component function is selected at each iteration in a cyclic or randomized manner. When the problem is constrained, the existing IG schemes (including projected IG, proximal IAG, and SAGA) require a projection step on…
We consider sum-type strongly convex optimization problem (first term) with smooth convex not proximal friendly composite (second term). We show that the complexity of this problem can be split into optimal number of incremental oracle calls for the first (sum-type) term and optimal number of oracle calls for the second (composite) term. Here under `optimal number' we mean estimate that corresponds to the well known lower bound in the absence of another term.
Incremental methods are widely utilized for solving finite-sum optimization problems in machine learning and signal processing. In this paper, we study a family of incremental methods -- including incremental subgradient, incremental proximal point, and incremental prox-linear methods -- for solving weakly convex optimization problems. Such a problem class covers many nonsmooth nonconvex instances that arise in engineering fields. We show that the three said incremental methods have an iteration complexity of $O(\varepsilon^{-4})$ for driving a natural stationarity measure to below $\varepsilo…
The University of Iowa（爱荷华大学）；University of California（加利福尼亚大学）；Pennsylvania State University（宾夕法尼亚州立大学）
Riemannian optimization has drawn a lot of attention due to its wide applications in practice. Riemannian stochastic first-order algorithms have been studied in the literature to solve large-scale machine learning problems over Riemannian manifolds. However, most of the existing Riemannian stochastic algorithms require the objective function to be differentiable, and they do not apply to the case where the objective function is nonsmooth. In this paper, we present two Riemannian stochastic proximal gradient methods for minimizing nonsmooth function over the Stiefel manifold. The two methods, n…
Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size
University of Utah（犹他大学）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）；Guangzhou Nanfang College（广州南方学院）；The Chinese University of Hong Kong（香港中文大学）
Institut de Mathématiques de Toulouse（图卢兹数学研究所）；CNRS（法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；Ecole Polytechnique（巴黎综合理工学院）
This paper introduces a novel algorithm, the Perturbed Proximal Preconditioned SPIDER algorithm (3P-SPIDER), designed to solve finite sum non-convex composite optimization. It is a stochastic Variable Metric Forward-Backward algorithm, which allows approximate preconditioned forward operator and uses a variable metric proximity operator as the backward operator; it also proposes a mini-batch strategy with variance reduction to address the finite sum setting. We show that 3P-SPIDER extends some Stochastic preconditioned Gradient Descent-based algorithms and some Incremental Expectation Maximiza…
School of Mathematical Sciences, Peking University（北京大学数学科学学院）
This paper studies the lower bound complexity for the optimization problem whose objective function is the average of $n$ individual smooth convex functions. We consider the algorithm which gets access to gradient and proximal oracle for each individual component. For the strongly-convex case, we prove such an algorithm can not reach an $\varepsilon$-suboptimal point in fewer than $Ω((n+\sqrt{κn})\log(1/\varepsilon))$ iterations, where $κ$ is the condition number of the objective function. This lower bound is tighter than previous results and perfectly matches the upper bound of the existing p…
We introduce a generic scheme for accelerating gradient-based optimization methods in the sense of Nesterov. The approach, called Catalyst, builds upon the inexact accelerated proximal point algorithm for minimizing a convex objective function, and consists of approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. One of the keys to achieve acceleration in theory and in practice is to solve these sub-problems with appropriate accuracy by using the right stopping criterion and the right warm-start strategy. We give practical guidelines to use Catalyst…
University of Cambridge（剑桥大学）；Queen Mary University of London（伦敦玛丽女王大学）
We present a generalisation of Rosenblatt's traditional perceptron learning algorithm to the class of proximal activation functions and demonstrate how this generalisation can be interpreted as an incremental gradient method applied to a novel energy function. This novel energy function is based on a generalised Bregman distance, for which the gradient with respect to the weights and biases does not require the differentiation of the activation function. The interpretation as an energy minimisation algorithm paves the way for many new algorithms, of which we explore a novel variant of the iter…
Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization
École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；INRIA（法国国家信息与自动化研究所）；ENS（巴黎高等师范学院）；MILA（蒙特利尔学习算法研究所）；DIRO（蒙特利尔大学计算机科学与运筹学系）；Université de Montréal（蒙特利尔大学）
