We analyse an iterative algorithm to minimize quadratic functions whose Hessian matrix $H$ is the expectation of a random symmetric $d\times d$ matrix. The algorithm is a variant of the stochastic variance reduced gradient (SVRG). In several applications, including least-squares regressions, ridge regressions, linear discriminant analysis and regularized linear discriminant analysis, the running time of each iteration is proportional to $d$. Under smoothness and convexity conditions, the algorithm has linear convergence. When applied to quadratic functions, our analysis improves the state-of-the-art performance of SVRG up to a logarithmic factor. Furthermore, for well-conditioned quadratic problems, our analysis improves the state-of-the-art running times of accelerated SVRG, and is better than the known matching lower bound, by a logarithmic factor. Our theoretical results are backed with numerical experiments.
Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimization
The Ohio State University（俄亥俄州立大学）；Duke University（杜克大学）
基于一阶技术SVRG和SARAH/SPIDER，最近分别设计了两类求解非凸优化问题的零阶随机算法。本文讨论了几个重要的问题，这些方法仍然是开放的。首先，所有现有的SVRG型零阶算法遭受更差的函数查询复杂度比零阶梯度下降（ZO-GD）或随机梯度下降（ZO-SGD）。在本文中，我们提出了一种新算法ZO-SVRG-Coord-Rand，并对Liu et al. 2018 b中提出的现有ZO-SVRG-Coord算法进行了新的分析，并表明ZO-SVRG-Coord-Rand和ZO-SVRG-Coord（在我们的新分析下）优于其他现有的SVRG型零阶方法以及ZO-GD和ZO-SGD。其次，现有的SPIDER型算法SPIDER-SZO（Fang et al. 2018）具有优越的理论性能，但在实践中会产生大量高斯随机变量以及$\sqrtε$级步长。在本文中，我们开发了一个新的算法ZO-SPIDER-Coord，这是免费的高斯变量的生成，并允许一个大的常数步长，同时保持相同的收敛速度和查询复杂度，我们进一步表明，ZO-SPIDER-Coord自动实现线性收敛速度，因为搜索器进入一个本地PL区域没有重新启动和算法修改。
