利用采样负动量直接加速SAGA
Direct Acceleration of SAGA using Sampled Negative Momentum
摘要:
方差缩减是一种简单而有效的方法,可以加速凸(或非凸)随机优化。在现有的方差缩减方法中,SVRG和SAGA采用无偏梯度估计,是近年来最流行的方差缩减方法。尽管SVRG的各种加速变体(例如,Katyusha和Acc-Prox-SVRG),但SAGA的直接加速仍然未知。在本文中,我们提出了一个直接加速的变体SAGA使用一种新的采样负动量(SSNM),它实现了最好的已知的预言复杂性强凸问题(与已知的强凸性参数)。因此,我们的工作填补了直接加速SAGA的空白。
英文摘要:
Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and are the most popular variance reduction methods in recent years. Although various accelerated variants of SVRG (e.g., Katyusha and Acc-Prox-SVRG) have been proposed, the direct acceleration of SAGA still remains unknown. In this paper, we propose a directly accelerated variant of SAGA using a novel Sampled Negative Momentum (SSNM), which achieves the best known oracle complexity for strongly convex problems (with known strong convexity parameter). Consequently, our work fills the void of directly accelerated SAGA.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。