随机拟梯度法:基于雅可比矩阵的方差缩减
Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketching
摘要:
我们发展了一个新的家庭的方差减少随机梯度下降法最小化的平均值的一个非常大的数量光滑函数。我们的方法--JacSketch--的动机是随机数值线性代数的新发展,并通过保持由各个函数的梯度组成的雅可比矩阵的随机估计来操作。在每次迭代中,JacSketch通过首先通过(廉价)草图绘制获得真实雅可比矩阵的随机线性测量,然后将先前的估计投影到线性矩阵方程的解空间上,该线性矩阵方程的解与测量一致,从而有效地更新雅可比矩阵。雅可比估计,然后用于计算方差减少无偏估计的梯度。我们的策略类似于拟牛顿法保持海森估计的方式,因此我们的方法可以被看作是一种随机拟梯度法。我们证明了,光滑和强凸函数,JacSketch收敛线性有意义的速度由一个单一的收敛定理,适用于一般草图。我们还提供了一个精致的收敛定理,适用于一类较小的草图。这使我们能够获得更清晰的复杂性结果的变种JacSketch的重要性采样。通过将我们的一般方法专门化到特定的草图策略,JacSketch简化为随机平均梯度(SAGA)方法,以及它的几个现有的和许多新的小批量,减少内存和重要性采样变体。我们使用重要性采样的SAGA的速率是该方法目前最知名的速率,解决了Schmidt et al(2015)的猜想。我们获得的小批量SAGA的速率也优于现有速率。
英文摘要:
We develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our method --JacSketch-- is motivated by novel developments in randomized numerical linear algebra, and operates by maintaining a stochastic estimate of a Jacobian matrix composed of the gradients of individual functions. In each iteration, JacSketch efficiently updates the Jacobian matrix by first obtaining a random linear measurement of the true Jacobian through (cheap) sketching, and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement. The Jacobian estimate is then used to compute a variance-reduced unbiased estimator of the gradient. Our strategy is analogous to the way quasi-Newton methods maintain an estimate of the Hessian, and hence our method can be seen as a stochastic quasi-gradient method. We prove that for smooth and strongly convex functions, JacSketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches. We also provide a refined convergence theorem which applies to a smaller class of sketches. This enables us to obtain sharper complexity results for variants of JacSketch with importance sampling. By specializing our general approach to specific sketching strategies, JacSketch reduces to the stochastic average gradient (SAGA) method, and several of its existing and many new minibatch, reduced memory, and importance sampling variants. Our rate for SAGA with importance sampling is the current best-known rate for this method, resolving a conjecture by Schmidt et al (2015). The rates we obtain for minibatch SAGA are also superior to existing rates.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。