arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1707.00424cs.LGcs.DCstat.ML
收藏

Parle:并行化随机梯度下降

Parle: parallelizing stochastic gradient descent

University of California, Los Angeles(加州大学洛杉矶分校);Bocconi University(博科尼大学);McGill University(麦吉尔大学)

Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, Ameet Talwalkar, Adam Oberman

摘要:

我们提出了一种名为Parle的新算法,用于深度网络的并行训练,其收敛速度比SGD的数据并行实现快2- 4倍,同时实现了显着改善的出错率,在包括CIFAR-10和CIFAR-100在内的几个基准测试中几乎是最先进的,而无需引入任何额外的超参数。我们利用平坦极小值现象,该现象已被证明可以改善深度网络的泛化误差。Parle不需要与参数服务器进行频繁的通信,而是在每个客户端上执行更多的计算,这使得它非常适合单机,多GPU设置和分布式实现。

英文摘要:

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑