分布式Optimization的统计预处理加速梯度法
Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization
摘要：
我们考虑分布式经验风险最小化的设置，其中多台机器并行计算梯度，集中式服务器更新模型参数。为了减少达到给定精度所需的通信次数，我们提出了一种\emph{preconditioned}加速梯度法，其中预处理是通过在服务器上解决二次采样数据集上的局部最优化问题来完成的。该方法的收敛速度取决于全局和局部损失函数之间的相对条件数的平方根。我们估计线性预测模型的相对条件数，通过研究\emph{uniform}浓度的Hessians在一个有界的域，这使我们能够得到改进的收敛速度现有的预处理梯度法和我们的加速方法。在真实世界数据集上的实验说明了在病态状态下加速的好处。
英文摘要：
We consider the setting of distributed empirical risk minimization where multiple machines compute the gradients in parallel and a centralized server updates the model parameters. In order to reduce the number of communications required to reach a given accuracy, we propose a \emph{preconditioned} accelerated gradient method where the preconditioning is done by solving a local optimization problem over a subsampled dataset at the server. The convergence rate of the method depends on the square root of the relative condition number between the global and local loss functions. We estimate the relative condition number for linear prediction models by studying \emph{uniform} concentration of the Hessians over a bounded domain, which allows us to derive improved convergence rates for existing preconditioned gradient methods and our accelerated method. Experiments on real-world datasets illustrate the benefits of acceleration in the ill-conditioned regime.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
