In this work, we propose an adaptive variation on the classical Heavy-ball method for convex quadratic minimization. The adaptivity crucially relies on so-called "Polyak step-sizes", which consists in using the knowledge of the optimal value of the optimization problem at hand instead of problem parameters such as a few eigenvalues of the Hessian of the problem. This method happens to also be equivalent to a variation of the classical conjugate gradient method, and thereby inherits many of its attractive features, including its finite-time convergence, instance optimality, and its worst-case convergence rates.
The classical gradient method with Polyak step-sizes is known to behave very well in situations in which it can be used, and the question of whether incorporating momentum in this method is possible and can improve the method itself appeared to be open. We provide a definitive answer to this question for minimizing convex quadratic functions, a arguably necessary first step for developing such methods in more general setups.
We study convergence of the trajectories of the Heavy Ball dynamical system, with constant damping coefficient, in the framework of convex and non-convex smooth optimization. By using the Polyak-Łojasiewicz condition, we derive new linear convergence rates for the associated trajectory, in terms of objective function values, without assuming uniqueness of the minimizer.
In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Nesterov later developed a globally accelerated version, Polyak's original algorithm remains simpler and more widely used in applications such as deep learning. Despite this popularity, the question of whether Heavy-ball is also globally accelerated or not has not been fully answered yet, and no convincing counterexample has been provided. This is largely due to the difficulty in finding an effective Lyapunov function: in…
According to a result due to B.T. Polyak, a mapping between Hilbert spaces, which is $C^{1,1}$ around a regular point, carries a ball centered at that point to a convex set, provided that the radius of the ball is small enough. The present paper considers the extension of such result to mappings defined on a certain subclass of uniformly convex Banach spaces. This enables one to extend to such setting a variational principle for constrained optimization problems, already observed in finite dimension, that establishes a convex behaviour for proper localizations of them. Further variational cons…
In this article a family of second order ODEs associated to inertial gradient descend is studied. These ODEs are widely used to build trajectories converging to a minimizer $x^*$ of a function $F$, possibly convex. This family includes the continuous version of the Nesterov inertial scheme and the continuous heavy ball method. Several damping parameters, not necessarily vanishing, and a perturbation term $g$ are thus considered. The damping parameter is linked to the inertia of the associated inertial scheme and the perturbation term $g$ is linked to the error that can be done on the gradient …
We derive an upper bound on the size of a ball such that the image of the ball under quadratic map is strongly convex and smooth. Our result is the best possible improvement of the analogous result by Polyak in the case of quadratic map. We also generalize the notion of the joint numerical range of m-tuple of matrices by adding vector-dependent inhomogeneous terms and provide a sufficient condition for its convexity.
This paper considers sufficient descent Riemannian conjugate gradient methods with line search algorithms. We propose two kinds of sufficient descent nonlinear conjugate gradient methods and prove these methods satisfy the sufficient descent condition even on Riemannian manifolds. One is the hybrid method combining the Fletcher-Reeves-type method with the Polak-Ribiere-Polyak-type method, and the other is the Hager-Zhang-type method, both of which are generalizations of those used in Euclidean space. Also, we generalize two kinds of line search algorithms that are widely used in Euclidean spac…
In the present paper, the following convexity principle is proved: any closed convex multifunction, which is metrically regular in a certain uniform sense near a given point, carries small balls centered at that point to convex sets, even if it is perturbed by adding C^{1,1} smooth mappings with controlled Lipschizian behaviour. This result, which is valid for mappings defined on a subclass of uniformly convex Banach spaces, can be regarded as a set-valued generalization of the Polyak convexity principle. The latter, indeed, can be derived as a special case of the former. Such an extension of …
In the present paper, the Polyak's principle, concerning convexity of the images of small balls through C1,1 mappings, is employed in the study of vector optimization problems. This leads to extend to such a context achievements of local programming, an approach to nonlinear optimization, due to B.T. Polyak, which consists in exploiting the benefits of the convex local behaviour of certain nonconvex problems. In doing so, solution existence and optimality conditions are established for localizations of vector optimization problems, whose data satisfy proper assumptions. Such results are subseq…
The stochastic inverse eigenvalue problem aims to reconstruct a stochastic matrix from its spectrum. While there exists a large literature on the existence of solutions for special settings, there are only few numerical solution methods available so far. Recently, Zhao et al. (2016) proposed a constrained optimization model on the manifold of so-called isospectral matrices and adapted a modified Polak-Ribière-Polyak conjugate gradient method to the geometry of this manifold. However, not every stochastic matrix is an isospectral one and the model from Zhao et al. is based on the assumption tha…
In this paper, we study the convergence properties of an algorithm that can be viewed as an interpolation between two gradient based optimization methods, Nesterov's acceleration method for strongly convex functions $(NAG$-$SC)$ and Polyak's heavy ball method. Recent Progress has been made on using High-Resolution ordinary differential equations (ODEs) to distinguish these two fundamentally different methods. The key difference between them can be attributed to the gradient correction term, which is reflected by the Hessian term in the High-Resolution ODE. Our goal is to understand how this te…
The goal is to modify the known method of mirror descent (MD), proposed by A.S. Nemirovsky and D.B. Yudin in 1979. The paper shows the idea of a new, so-called inertial MD method with the example of a deterministic optimization problem in continuous time. In particular, in the Euclidean case, the heavy ball method by B.T. Polyak is realized. It is noted that the new method does not use additional averaging. A discrete algorithm of inertial MD is described. The theorem on the upper bound on the error in the objective function is proved.
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a particular second order differential equation with friction. Such connections with continuous-time dynamical systems have been instrumental in demystifying acceleration phenomena in optimization. Here we study structure-preserving discretizations for a certain class of dissipative (conformal) Hamiltonian systems, allowing us to analyze the symplectic structure of both Nesterov and…
Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
Florida International University（佛罗里达国际大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
CNRS（法国国家科学研究中心）；ENS（巴黎高等师范学院）；PSL University（巴黎文理研究大学）；CREST（经济与统计研究中心）；ENSAE（巴黎高等经济商业学院）；Télécom ParisTech（巴黎高等电信学院）；IP Paris（巴黎理工学院）；Facebook AI Research（脸书人工智能研究院）
We study stochastic gradient descent (SGD) and the stochastic heavy ball method (SHB, otherwise known as the momentum method) for the general stochastic approximation problem. For SGD, in the convex and smooth setting, we provide the first \emph{almost sure} asymptotic convergence \emph{rates} for a weighted average of the iterates . More precisely, we show that the convergence rate of the function values is arbitrarily close to $o(1/\sqrt{k})$, and is exactly $o(1/k)$ in the so-called overparametrized case. We show that these results still hold when using stochastic line search and stochastic…
Many problems arise in computational biology can be reduced to the minimization of energy function, that determines on the geometry of considered molecule. The solution of this problem allows in particular to solve folding and docking problems in structural biology. For the small molecules this problem is well solved. But for the large molecules ($10^4$ atoms and more) this is still an open problem. In this work we consider energy minimization problem (OPLS force field) for the large molecules but with good enough initial (starting) point. In the paper one can find a biological explanation of …
Momentum-based gradient descent methods for Lie groups
Departament de Matemàtiques, Universitat de València（瓦伦西亚大学数学系）；Departamento de Matemática Aplicada, Ciencia e Ingeniería de los Materiales y Tecnología Electrónica, Universidad Rey Juan Carlos（雷乌恩卡洛斯大学应用数学系）；Instituto de Ciencias Matemáticas (CSIC-UAM-UC3M-UCM)（数学科学研究所）；Departamento de Matemáticas, Universidad de Córdoba（科尔多瓦大学数学系）
University of Freiburg（弗莱堡大学）；BIOSS Centre for Biological Signalling Studies（BIOSS生物信号研究中心）；Graz University of Technology（格拉茨工业大学）
In this paper we study an algorithm for solving a minimization problem composed of a differentiable (possibly non-convex) and a convex (possibly non-differentiable) function. The algorithm iPiano combines forward-backward splitting with an inertial force. It can be seen as a non-smooth split version of the Heavy-ball method from Polyak. A rigorous analysis of the algorithm for the proposed class of problems yields global convergence of the function values and the arguments. This makes the algorithm robust for usage on non-convex problems. The convergence result is obtained based on the \KL ine…
Accelerated gradient (AG) methods are breakthroughs in convex optimization, improving the convergence rate of the gradient descent method for optimization with smooth functions. However, the analysis of AG methods for non-convex optimization is still limited. It remains an open question whether AG methods from convex optimization can accelerate the convergence of the gradient descent method for finding local minimum of non-convex optimization problems. This paper provides an affirmative answer to this question. In particular, we analyze two renowned variants of AG methods (namely Polyak's Heav…
