arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1805.07786math.OC
收藏

打破跨度假设可快速实现最小化

Breaking the Span Assumption Yields Fast Finite-Sum Minimization

Robert Hannah, Yanli Liu, Daniel O'Connor, Wotao Yin

摘要:

在本文中,我们表明,SVRG和SARAH可以修改,从根本上比所有其他标准算法,最小化的总和$n$光滑函数,如SAGA,SAG,SDCA,SDCA没有对偶。大多数有限和算法遵循我们所谓的“跨度假设”:它们的更新是在以随机IID方式选择的分量梯度序列的跨度中。在条件数为$κ=\mathcal{O}(n)$的大数据体系中,跨度假设防止算法在少于$n\ln(1/ε)$的迭代中收敛到精确度为$ε$的近似解。SVRG和SARAH不遵循跨度假设,因为它们是用全梯度和分量梯度信息的混合更新的。我们表明,正因为如此,他们可以高达$Ω(1+(\ln(n/κ))_+)$倍的速度。特别地,为了获得$κ=n^β$和$α,β\in(0,1)$的准确度$ε= 1/n^α$,修改的SVRG需要$\mathcal{O}(n)$迭代,而遵循跨度假设的算法需要$\mathcal{O}(n\ln(n))$迭代。此外,我们提出了下界的结果表明,这种加速是最佳的,并提供分析,以帮助解释为什么存在这种加速。随着理解跨度假设是有限和算法的弱点,未来的工作可能会有目的地利用这一点,在大数据领域产生更快的算法。

英文摘要:

In this paper, we show that SVRG and SARAH can be modified to be fundamentally faster than all of the other standard algorithms that minimize the sum of $n$ smooth functions, such as SAGA, SAG, SDCA, and SDCA without duality. Most finite sum algorithms follow what we call the "span assumption": Their updates are in the span of a sequence of component gradients chosen in a random IID fashion. In the big data regime, where the condition number $κ=\mathcal{O}(n)$, the span assumption prevents algorithms from converging to an approximate solution of accuracy $ε$ in less than $n\ln(1/ε)$ iterations. SVRG and SARAH do not follow the span assumption since they are updated with a hybrid of full-gradient and component-gradient information. We show that because of this, they can be up to $Ω(1+(\ln(n/κ))_+)$ times faster. In particular, to obtain an accuracy $ε= 1/n^α$ for $κ=n^β$ and $α,β\in(0,1)$, modified SVRG requires $\mathcal{O}(n)$ iterations, whereas algorithms that follow the span assumption require $\mathcal{O}(n\ln(n))$ iterations. Moreover, we present lower bound results that show this speedup is optimal, and provide analysis to help explain why this speedup exists. With the understanding that the span assumption is a point of weakness of finite sum algorithms, future work may purposefully exploit this to yield even faster algorithms in the big data regime.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑