Accelerated and nonaccelerated stochastic gradient descent with model conception

Accelerated and nonaccelerated stochastic gradient descent with inexact model

In this paper, we propose a new way to obtain optimal convergence rates for smooth stochastic (strong) convex optimization tasks. Our approach is based on results for optimization tasks where gradients have nonrandom noise. In contrast to previously known results, we extend our idea to the inexact model conception.

About accelerated randomized methods

We show how one can obtain nonaccelerated randomized coordinate descent method (Yu. Nesterov, 2010) and nonaccelerated method of randomization of sum-type functional (Le Roux-Schmidt-Bach, 2012) from the optimal method for the stochastic optimization problem (SIGMA, Devolder-Glineur-Nesterov-Dvurechensky-Gasnikov, 2014). The main trick is a special restart technique. We considered this trick to be usefull in others contexts. We consider only strongly convex case. We show that accelerated variants of this methods seems to be nontrivial in this context. That is, it is hard (perhaps impossible) t…

Accelerated Coordinate Descent with Arbitrary Sampling and Best Rates for Minibatches

Accelerated coordinate descent is a widely popular optimization algorithm due to its efficiency on large-dimensional problems. It achieves state-of-the-art complexity on an important class of empirical risk minimization problems. In this paper we design and analyze an accelerated coordinate descent (ACD) method which in each iteration updates a random subset of coordinates according to an arbitrary but fixed probability law, which is a parameter of the method. If all coordinates are updated in each iteration, our method reduces to the classical accelerated gradient descent method AGD of Nester…

中心弱离散黎曼梯度：黎曼优化的统一框架

Centered Weak Discrete Riemannian Gradients: A Unified Framework for Riemannian Optimization

提出中心弱离散黎曼梯度（c-wDRG）框架，统一分析黎曼流形优化方法，推导收敛结果并开发加速方案，为相关方法提供显式步长条件与收敛速率。