arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.13591cs.LGcs.DCstat.ML
收藏

并行学习的随机块对角预处理算法

Randomized Block-Diagonal Preconditioning for Parallel Learning

University of California, Berkeley(加州大学伯克利分校);ETH Zürich(苏黎世联邦理工学院)

Celestine Mendler-Dünner, Aurelien Lucchi

摘要:

我们研究预条件梯度优化方法的预条件矩阵具有块对角形式。这种结构约束的优点是更新计算是块可分离的,并且可以跨多个独立任务并行化。我们的主要贡献是证明,这些方法的收敛性可以显着提高的随机化技术,对应于重新分区坐标跨任务的优化过程中。我们提供了一个理论分析,准确地描述了预期的收敛收益的重新划分和验证我们的研究结果经验上的各种传统的机器学习任务。从实现的角度来看,块可分离模型非常适合并行化,并且当共享内存可用时,可以在现有方法之上非常有效地实现随机化以提高收敛性。

英文摘要:

We study preconditioned gradient-based optimization methods where the preconditioning matrix has block-diagonal form. Such a structural constraint comes with the advantage that the update computation is block-separable and can be parallelized across multiple independent tasks. Our main contribution is to demonstrate that the convergence of these methods can significantly be improved by a randomization technique which corresponds to repartitioning coordinates across tasks during the optimization procedure. We provide a theoretical analysis that accurately characterizes the expected convergence gains of repartitioning and validate our findings empirically on various traditional machine learning tasks. From an implementation perspective, block-separable models are well suited for parallelization and, when shared memory is available, randomization can be implemented on top of existing methods very efficiently to improve convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑