We present a new accelerated gradient-based method for solving smooth unconstrained optimization problems. The goal is to embed a heavy-ball type of momentum into the Fast Gradient Method (FGM). For this purpose, we devise a generalization of the estimating sequences, which allows for encoding any form of information about the cost function that can aid in further accelerating the minimization process. In the black box framework, we propose a construction for the generalized estimating sequences, which is obtained by exploiting the history of the previously constructed estimating functions. From the viewpoint of efficiency estimates, we prove that the lower bound on the number of iterations for the proposed method is $\mathcal{O} \left(\sqrt{\fracκ{2}}\right)$. Our theoretical results are further corroborated by extensive numerical experiments on various types of optimization problems, often dealt within signal processing. Both synthetic and real-world datasets are utilized to demonstrate the efficiency of our proposed method in terms of decreasing the distance to the optimal solution, as well as in terms of decreasing the norm of the gradient.
This paper establishes global convergence and provides global bounds of the convergence rate of the Heavy-ball method for convex optimization problems. When the objective function has Lipschitz-continuous gradient, we show that the Cesaro average of the iterates converges to the optimum at a rate of $O(1/k)$ where k is the number of iterations. When the objective function is also strongly convex, we prove that the Heavy-ball iterates converge linearly to the unique optimum.
We study convergence of the trajectories of the Heavy Ball dynamical system, with constant damping coefficient, in the framework of convex and non-convex smooth optimization. By using the Polyak-Łojasiewicz condition, we derive new linear convergence rates for the associated trajectory, in terms of objective function values, without assuming uniqueness of the minimizer.
Heavy-Ball method (HB) is known for its simplicity in implementation and practical efficiency. However, as with other momentum methods, it has non-monotone behavior, and for optimal parameters, the method suffers from the so-called peak effect. To address this issue, in this paper, we consider an averaged version of Heavy-Ball method (AHB). We show that for quadratic problems AHB has a smaller maximal deviation from the solution than HB. Moreover, for general convex and strongly convex functions, we prove non-accelerated rates of global convergence of AHB and its weighted version. We conduct s…
In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Nesterov later developed a globally accelerated version, Polyak's original algorithm remains simpler and more widely used in applications such as deep learning. Despite this popularity, the question of whether Heavy-ball is also globally accelerated or not has not been fully answered yet, and no convincing counterexample has been provided. This is largely due to the difficulty in finding an effective Lyapunov function: in…
Momentum first-order optimization methods are the workhorses in various optimization tasks, e.g., in the training of deep neural networks. Recently, Lucas et al. (2019) proposed a method called Aggregated Heavy-Ball (AggHB) that uses multiple momentum vectors corresponding to different momentum parameters and averages these vectors to compute the update direction at each iteration. Lucas et al. (2019) show that AggHB is more stable than the classical Heavy-Ball method even with large momentum parameters and performs well in practice. However, the method was analyzed only for quadratic objectiv…
We develop a distributed algorithm for convex Empirical Risk Minimization, the problem of minimizing large but finite sum of convex functions over networks. The proposed algorithm is derived from directly discretizing the second-order heavy-ball differential equation and results in an accelerated convergence rate, i.e, faster than distributed gradient descent-based methods for strongly convex objectives that may not be smooth. Notably, we achieve acceleration without resorting to the well-known Nesterov's momentum approach. We provide numerical experiments and contrast the proposed method with…
We focus on the solutions of second-order stable linear difference equations and demonstrate that their behavior can be non-monotone and exhibit peak effects depending on initial conditions. The results are applied to the analysis of the accelerated unconstrained optimization method -- the Heavy Ball method. We explain non-standard behavior of the method discovered in practical applications. In addition, such non-monotonicity complicates the correct choice of the parameters in optimization methods. We propose to overcome this difficulty by introducing new Lyapunov function which should decreas…
Sharp Initialization Guarantees for the Heavy-Ball Method
Xi’an Jiaotong University（西安交通大学）；Hong Kong Polytechnic University（香港理工大学）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）；A*STAR（新加坡科技研究局）
We present a stochastic variance-reduced heavy ball power iteration algorithm for solving PCA and provide a convergence analysis for it. The algorithm is an extension of heavy ball power iteration, incorporating a step size so that progress can be controlled depending on the magnitude of the variance of stochastic gradients. The algorithm works with any size of the mini-batch, and if the step size is appropriately chosen, it attains global linear convergence to the first eigenvector of the covariance matrix in expectation. The global linear convergence result in expectation is analogous to tho…
A local convergence result for abstract descent methods is proved. The sequence of iterates is attracted by a local (or global) minimum, stays in its neighborhood and converges within this neighborhood. This result allows algorithms to exploit local properties of the objective function. In particular, the abstract theory in this paper applies to the inertial forward--backward splitting method: iPiano---a generalization of the Heavy-ball method. Moreover, it reveals an equivalence between iPiano and inertial averaged/alternating proximal minimization and projection methods. Key for this equival…
We investigate the long time behavior of solutions to the differential equation $\ddot{x}(t)+\frac{c}{\left( t+1\right) ^α}\dot{x}(t)+\nabla Φ\left( x(t)\right) =g(t),~t\geq0, $ where $c$ is nonnegative constant, $α\in\lbrack0,1[,$ $Φ$ is a $C^{1}$ convex function on a Hilbert space $\mathcal{H}$ and $g\in L^{1} (0,+\infty;\mathcal{H}).$ We obtain sufficient conditions on the source term $g(t)$ ensuring the weak or the strong convergence of any trajectory $x(t)$ as $t\rightarrow+\infty$ to a minimizer of the function $Φ$ if one exists.
Conditional gradient, aka Frank Wolfe (FW) algorithms, have well-documented merits in machine learning and signal processing applications. Unlike projection-based methods, momentum cannot improve the convergence rate of FW, in general. This limitation motivates the present work, which deals with heavy ball momentum, and its impact to FW. Specifically, it is established that heavy ball offers a unifying perspective on the primal-dual (PD) convergence, and enjoys a tighter per iteration PD error rate, for multiple choices of step sizes, where PD error can serve as the stopping criterion in pract…
In this work, we propose an adaptive variation on the classical Heavy-ball method for convex quadratic minimization. The adaptivity crucially relies on so-called "Polyak step-sizes", which consists in using the knowledge of the optimal value of the optimization problem at hand instead of problem parameters such as a few eigenvalues of the Hessian of the problem. This method happens to also be equivalent to a variation of the classical conjugate gradient method, and thereby inherits many of its attractive features, including its finite-time convergence, instance optimality, and its worst-case c…
We study distributed optimization to minimize a global objective that is a sum of smooth and strongly-convex local cost functions. Recently, several algorithms over undirected and directed graphs have been proposed that use a gradient tracking method to achieve linear convergence to the global minimizer. However, a connection between these different approaches has been unclear. In this paper, we first show that many of the existing first-order algorithms are in fact related with a simple state transformation, at the heart of which lies the $\mathcal{AB}$ algorithm. We then describe \textit{dis…
National University of Defense Technology（国防科技大学）；Hunan University（湖南大学）
Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoid saddle points? The answer is yes! Direct using the existing proof technique for the heavy-ball algorithms is hard due to that each iteration of the heavy-ball algorithm consists of current and last points. It is impossible to formulate the algorithms as iteration like xk+1= g(x…
In this paper, we revisit the convergence of the Heavy-ball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature. We extend our results to multi-block version of the algorithm with both the cyclic and stochastic update r…
Abs-Smooth Frank-Wolfe Method: Primal-Dual Analysis, Heavy Ball Momentum, and Inexact Oracles
Institut für Mathematik, Humboldt-Universität zu Berlin（柏林洪堡大学数学研究所）；Zuse Institute Berlin（柏林祖斯研究所）；Institut für Mathematik, Technische Universität Berlin（柏林工业大学数学研究所）
In this article a family of second order ODEs associated to inertial gradient descend is studied. These ODEs are widely used to build trajectories converging to a minimizer $x^*$ of a function $F$, possibly convex. This family includes the continuous version of the Nesterov inertial scheme and the continuous heavy ball method. Several damping parameters, not necessarily vanishing, and a perturbation term $g$ are thus considered. The damping parameter is linked to the inertia of the associated inertial scheme and the perturbation term $g$ is linked to the error that can be done on the gradient …
In this letter we revisit the famous heavy ball method and study its global convergence for a class of non-convex problems with sector-bounded gradient. We characterize the parameters that render the method globally convergent and yield the best $R$-convergence factor. We show that for this family of functions, this convergence factor is superior to the factor obtained from the triple momentum method.
The recently proposed Sampling Kaczmarz Motzkin (SKM) algorithm performs well in comparison with the state-of-the-art methods in solving large-scale Linear Feasibility (LF) problems. To explore the concept of momentum in the context of solving LF problems, in this work, we propose a momentum induced algorithm called Momentum Sampling Kaczmarz Motzkin (MSKM). The MSKM algorithm is developed by integrating the heavy ball momentum to the SKM algorithm. We provide a rigorous convergence analysis of the proposed MSKM algorithm from which we obtain convergence results of several Kaczmarz type method…
On the fast convergence of minibatch heavy ball momentum
Operations Research and Industrial Engineering（运筹学与工业工程）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Mathematics（数学）；New York University（纽约大学）；Computer Science and Engineering（计算机科学与工程）
It is well known that the finite step-size ($h$) in Gradient Descent (GD) implicitly regularizes solutions to flatter minima. A natural question to ask is "Does the momentum parameter $β$ play a role in implicit regularization in Heavy-ball (H.B) momentum accelerated gradient descent (GD+M)?". To answer this question, first, we show that the discrete H.B momentum update (GD+M) follows a continuous trajectory induced by a modified loss, which consists of an original loss and an implicit regularizer. Then, we show that this implicit regularizer for (GD+M) is stronger than that of (GD) by factor …
In this paper, we study the convergence properties of an algorithm that can be viewed as an interpolation between two gradient based optimization methods, Nesterov's acceleration method for strongly convex functions $(NAG$-$SC)$ and Polyak's heavy ball method. Recent Progress has been made on using High-Resolution ordinary differential equations (ODEs) to distinguish these two fundamentally different methods. The key difference between them can be attributed to the gradient correction term, which is reflected by the Hessian term in the High-Resolution ODE. Our goal is to understand how this te…
National University of Defense Technology（国防科技大学）；Tsinghua University（清华大学）；University of Utah（犹他大学）
Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a uniform hyperparameter, which relies on excessive tuning. Moreover, the calibrated fixed hyperparameter may not lead to optimal performance. In this paper, to eliminate the effort for tuning the momentum-related hyperparameter, we propose a new adaptive momentum inspired by the optimal choice of the heavy ball momentum for quadratic optimization. Our proposed adaptive heavy ball momentum can improve stochastic gradient de…
Stochastic dual coordinate descent with adaptive heavy ball momentum for linearly constrained convex optimization
School of Mathematical Sciences, Beihang University（北京航空航天大学数学科学学院）；LMIB of the Ministry of Education, School of Mathematical Sciences, Beihang University（教育部数学线性代数重点实验室，北京航空航天大学数学科学学院）
CNRS（法国国家科学研究中心）；ENS（巴黎高等师范学院）；PSL University（巴黎文理研究大学）；CREST（经济与统计研究中心）；ENSAE（巴黎高等经济商业学院）；Télécom ParisTech（巴黎高等电信学院）；IP Paris（巴黎理工学院）；Facebook AI Research（脸书人工智能研究院）
We study stochastic gradient descent (SGD) and the stochastic heavy ball method (SHB, otherwise known as the momentum method) for the general stochastic approximation problem. For SGD, in the convex and smooth setting, we provide the first \emph{almost sure} asymptotic convergence \emph{rates} for a weighted average of the iterates . More precisely, we show that the convergence rate of the function values is arbitrarily close to $o(1/\sqrt{k})$, and is exactly $o(1/k)$ in the so-called overparametrized case. We show that these results still hold when using stochastic line search and stochastic…
Accelerated Gossip via Stochastic Heavy Ball Method
School of Mathematics, KAUST, Saudi Arabia（卡士塔大学数学学院，沙特阿拉伯）；The University of Edinburgh（爱丁堡大学）；The University of Edinburgh, United Kingdom（英国爱丁堡大学）；Edinburgh, Scotland, UK（苏格兰爱丁堡，英国）；MIPT, Russia（莫斯科国立信息安全大学，俄罗斯）
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dynamical systems have been instrumental in demystifying acceleration phenomena in optimization. Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyze the symplectic structure of both Nesterov and…
