arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1702.06269cs.LG
收藏

基于Minibatch-Prox的内存和通信高效分布式随机优化

Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox

University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田工业大学)

Jialei Wang, Weiran Wang, Nathan Srebro

摘要:

我们提出并分析了一种分布式随机优化的方法,它是统计上最优的,并实现了近线性的加速比(对数因子)。我们的方法允许通信内存的权衡,无论是对数通信,但线性内存,或多项式通信和相应的多项式减少所需的内存。这种通信-内存权衡是通过小批量-接近迭代(小批量被动-主动更新)实现的,其中在每次迭代中解决小批量上的子问题。我们提供了一种新的分析,这样的小批量prox程序,实现了统计最优速率,而不管小批量的大小和平滑度,从而显着改善以前的工作。

英文摘要:

We present and analyze an approach for distributed stochastic optimization which is statistically optimal and achieves near-linear speedups (up to logarithmic factors). Our approach allows a communication-memory tradeoff, with either logarithmic communication but linear memory, or polynomial communication and a corresponding polynomial reduction in required memory. This communication-memory tradeoff is achieved through minibatch-prox iterations (minibatch passive-aggressive updates), where a subproblem on a minibatch is solved at each iteration. We provide a novel analysis for such a minibatch-prox procedure which achieves the statistical optimal rate regardless of minibatch size and smoothness, thus significantly improving on prior work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑