Variance reduction techniques like SVRG provide simple and fast algorithms for optimizing a convex finite-sum objective. For nonconvex objectives, these techniques can also find a first-order stationary point (with small gradient). However, in nonconvex optimization it is often crucial to find a second-order stationary point (with small gradient and almost PSD hessian). In this paper, we show that Stabilized SVRG (a simple variant of SVRG) can find an $ε$-second-order stationary point using only $\widetilde{O}(n^{2/3}/ε^2+n/ε^{1.5})$ stochastic gradients. To our best knowledge, this is the first second-order guarantee for a simple variant of SVRG. The running time almost matches the known guarantees for finding $ε$-first-order stationary points.
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
The Ohio State University（俄亥俄州立大学）；Duke University（杜克大学）
基于一阶技术SVRG和SARAH/SPIDER，最近分别设计了两类求解非凸优化问题的零阶随机算法。本文讨论了几个重要的问题，这些方法仍然是开放的。首先，所有现有的SVRG型零阶算法遭受更差的函数查询复杂度比零阶梯度下降（ZO-GD）或随机梯度下降（ZO-SGD）。在本文中，我们提出了一种新算法ZO-SVRG-Coord-Rand，并对Liu et al. 2018 b中提出的现有ZO-SVRG-Coord算法进行了新的分析，并表明ZO-SVRG-Coord-Rand和ZO-SVRG-Coord（在我们的新分析下）优于其他现有的SVRG型零阶方法以及ZO-GD和ZO-SGD。其次，现有的SPIDER型算法SPIDER-SZO（Fang et al. 2018）具有优越的理论性能，但在实践中会产生大量高斯随机变量以及$\sqrtε$级步长。在本文中，我们开发了一个新的算法ZO-SPIDER-Coord，这是免费的高斯变量的生成，并允许一个大的常数步长，同时保持相同的收敛速度和查询复杂度，我们进一步表明，ZO-SPIDER-Coord自动实现线性收敛速度，因为搜索器进入一个本地PL区域没有重新启动和算法修改。
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each iteration, our algorithm uses $K+1$ nested reference points to build a semi-stochastic gradient to further reduce its variance in each iteration. For smooth nonconvex functions, the proposed algorith…
Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
MIT-IBM Watson AI Lab（麻省理工学院-IBM Watson人工智能实验室）；IBM Research（IBM研究）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Katyusha X: Practical Momentum Method for Stochastic Sum-of-Nonconvex Optimization
Microsoft Research AI（微软研究院AI）
最小化非凸函数之和的问题（即，凸函数（即非凸函数的平均值）在机器学习中变得越来越重要，并且是PCA、SVD、正则化牛顿法、加速非凸优化等的核心机制。我们展示了如何可证明地获得一个加速的随机算法最小化的总和的非凸函数，由$\textit{adding one additional line}$著名的SVRG方法。这条线对应于动量，并显示了如何直接将动量应用于非凸函数之和的有限和随机最小化。作为一个副作用，我们的方法享有线性并行加速使用小批量。
King Abdullah University of Science Technology（阿卜杜拉国王科技大学）
We propose ZeroSARAH -- a novel variant of the variance-reduced method SARAH (Nguyen et al., 2017) -- for minimizing the average of a large number of nonconvex functions $\frac{1}{n}\sum_{i=1}^{n}f_i(x)$. To the best of our knowledge, in this nonconvex finite-sum regime, all existing variance-reduced methods, including SARAH, SVRG, SAGA and their variants, need to compute the full gradient over all $n$ data samples at the initial point $x^0$, and then periodically compute the full gradient once every few iterations (for SVRG, SARAH and their variants). Note that SVRG, SAGA and their variants t…
Stochastic Non-convex Ordinal Embedding with Stabilized Barzilai-Borwein Step Size
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Jiangxi Normal University（江西师范大学）；Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯人工智能实验室）
Low-rank inductive matrix completion (IMC) is currently widely used in IoT data completion, recommendation systems, and so on, as the side information in IMC has demonstrated great potential in reducing sample point remains a major obstacle for the convergence of the nonconvex solutions to IMC. What's more, carefully choosing the initial solution alone does not usually help remove the saddle points. To address this problem, we propose a stocastic variance reduction gradient-based algorithm called LRSVRG-IMC. LRSVRG-IMC can escape from the saddle points under various low-rank and sparse conditi…
We present a unified framework to analyze the global convergence of Langevin dynamics based algorithms for nonconvex finite-sum optimization with $n$ component functions. At the core of our analysis is a direct analysis of the ergodicity of the numerical approximations to Langevin dynamics, which leads to faster convergence rates. Specifically, we show that gradient Langevin dynamics (GLD) and stochastic gradient Langevin dynamics (SGLD) converge to the almost minimizer within $\tilde O\big(nd/(λε) \big)$ and $\tilde O\big(d^7/(λ^5ε^5) \big)$ stochastic gradient evaluations respectively, where…
ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization
IBM Research（IBM研究院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
我们提出了一个新的随机一阶算法框架来解决随机复合非凸最优化问题，包括有限和和期望设置。我们的算法依赖于（Nguyen et al，2017）中引入的SARAH估计器，由两个步骤组成：近似梯度和平均步骤，使它们不同于现有的非凸近似型算法。该算法只需要非凸目标项的平均光滑性假设和附加的有界方差假设，如果应用于期望问题。它们可以使用恒定和自适应步长，同时允许单个样品和小批量。在所有这些情况下，我们证明了我们的算法可以实现最知名的复杂性界限。我们的方法的一个关键步骤是新的常数和自适应步长，有助于实现所需的复杂度范围，同时提高实际性能。我们的恒定步长是远远大于现有的方法，包括近端SVRG计划在单样本的情况下。我们还指定的算法的非复合材料的情况下，涵盖现有的国家的最先进的复杂性界限。我们的更新还允许在步长和小批量大小之间进行权衡，以提高性能。我们测试两个复合非凸问题和神经网络使用几个著名的数据集上提出的算法。
TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction
Department of Mathematics, University of California, Berkeley, CA, USA（加州大学伯克利分校数学系）；Boston University, MA, USA（波士顿大学）；Department of IEOR, University of California, Berkeley, CA, USA（加州大学伯克利分校工业工程与运筹学系）
School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）；Institute of Artificial Intelligence, Jiangxi Normal University（江西师范大学人工智能研究院）；Institute of Computing, Chinese Academy of Sciences（中国科学院计算技术研究所）；Chemical and Biomolecular Engineering, Hong Kong University of Science and Technology（香港科技大学化学与生物分子工程系）
The low-rank stochastic semidefinite optimization has attracted rising attention due to its wide range of applications. The nonconvex reformulation based on the low-rank factorization, significantly improves the computational efficiency but brings some new challenge to the analysis. The stochastic variance reduced gradient (SVRG) method has been regarded as one of the most effective methods. SVRG in general consists of two loops, where a reference full gradient is first evaluated in the outer loop and then used to yield a variance reduced estimate of the current gradient in the inner loop. Two…
Computational Science Initiative（计算科学计划）；Brookhaven National Laboratory（布鲁克海文国家实验室）
This work examines the convergence of stochastic gradient-based optimization algorithms that use early stopping based on a validation function. The form of early stopping we consider is that optimization terminates when the norm of the gradient of a validation function falls below a threshold. We derive conditions that guarantee this stopping rule is well-defined, and provide bounds on the expected number of iterations and gradient evaluations needed to meet this criterion. The guarantee accounts for the distance between the training and validation sets, measured with the Wasserstein distance.…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学（KAUST））
In this paper, we study the performance of a large family of SGD variants in the smooth nonconvex regime. To this end, we propose a generic and flexible assumption capable of accurate modeling of the second moment of the stochastic gradient. Our assumption is satisfied by a large number of specific variants of SGD in the literature, including SGD with arbitrary sampling, SGD with compressed gradients, and a wide variety of variance-reduced SGD methods such as SVRG and SAGA. We provide a single convergence analysis for all methods that satisfy the proposed unified assumption, thereby offering a…
A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems
Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Department of ISE, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校工业与系统工程系）
Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）；The Chinese University of Hong Kong（香港中文大学）；Columbia University（哥伦比亚大学）；Tencent AI Lab（腾讯人工智能实验室）
University of Pittsburgh（匹兹堡大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；MIIT Key Laboratory of Pattern Analysis & Machine Intelligence（工业和信息化部模式分析与机器智能重点实验室）；JD Finance America Corporation（京东金融美国公司）
In this paper, we propose a faster stochastic alternating direction method of multipliers (ADMM) for nonconvex optimization by using a new stochastic path-integrated differential estimator (SPIDER), called as SPIDER-ADMM. Moreover, we prove that the SPIDER-ADMM achieves a record-breaking incremental first-order oracle (IFO) complexity of $\mathcal{O}(n+n^{1/2}ε^{-1})$ for finding an $ε$-approximate stationary point, which improves the deterministic ADMM by a factor $\mathcal{O}(n^{1/2})$, where $n$ denotes the sample size. As one of major contribution of this paper, we provide a new theoretica…
Fast Stochastic Ordinal Embedding with Variance Reduction and Adaptive Step Size
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Jiangxi Normal University（江西师范大学）；Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
The Ohio State University（俄亥俄州立大学）；University of Utah（犹他大学）；Southern University of Science and Technology（南方科技大学）
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-size adaption rule or exploit the information along optimization path via additional backtracking and condition verification steps. In this paper, we propose a novel scheme, which eliminates backtracking line search but still exploits the information along optimization path by adapt…
Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size
University of Utah（犹他大学）；Beijing Normal-Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）；Guangzhou Nanfang College（广州南方学院）；The Chinese University of Hong Kong（香港中文大学）
