高效统计预处理分布式Optimization的超快速二阶局部求解器
Hyperfast Second-Order Local Solvers for Efficient Statistically Preconditioned Distributed Optimization
摘要：
统计预处理使分布式大规模经验风险最小化问题的快速方法成为可能。在这种方法中，多个工作节点并行计算梯度，然后由中心节点通过解决辅助（预处理）较小规模的最优化问题来更新参数。最近提出的统计预处理加速梯度（SPAG）方法的复杂性界限优于其他此类算法，但需要一个精确解，计算密集型辅助最优化问题在每次迭代。在本文中，我们提出了一个不精确的SPAG（InSPAG），并明确地描述了相应的辅助子问题需要解决的精度，以保证相同的收敛速度的精确方法。我们建立我们的结果，首先发展一个不精确的自适应加速Bregman近似梯度法一般最优化问题下的相对光滑性和强凸性假设，这可能是独立的利益。此外，我们探讨的InSPAG算法假设Lipschitz三阶导数和强凸性的辅助问题的性质。对于这样的问题类，我们开发了一个线性收敛的超快二阶方法和估计的总复杂度的InSPAG方法与超快辅助问题求解器。最后，我们通过对逻辑斯谛回归模型进行大规模数值实验，说明了所提出方法的实际效率。据我们所知，这些是第一次实施高阶方法对大规模问题的实证结果，我们的工作与数据的维度是~ $3$百万的顺序，和样本个数是~ $700$百万。
英文摘要：
Statistical preconditioning enables fast methods for distributed large-scale empirical risk minimization problems. In this approach, multiple worker nodes compute gradients in parallel, which are then used by the central node to update the parameter by solving an auxiliary (preconditioned) smaller-scale optimization problem. The recently proposed Statistically Preconditioned Accelerated Gradient (SPAG) method has complexity bounds superior to other such algorithms but requires an exact solution for computationally intensive auxiliary optimization problems at every iteration. In this paper, we propose an Inexact SPAG (InSPAG) and explicitly characterize the accuracy by which the corresponding auxiliary subproblem needs to be solved to guarantee the same convergence rate as the exact method. We build our results by first developing an inexact adaptive accelerated Bregman proximal gradient method for general optimization problems under relative smoothness and strong convexity assumptions, which may be of independent interest. Moreover, we explore the properties of the auxiliary problem in the InSPAG algorithm assuming Lipschitz third-order derivatives and strong convexity. For such problem class, we develop a linearly convergent Hyperfast second-order method and estimate the total complexity of the InSPAG method with hyperfast auxiliary problem solver. Finally, we illustrate the proposed method's practical efficiency by performing large-scale numerical experiments on logistic regression models. To the best of our knowledge, these are the first empirical results on implementing high-order methods on large-scale problems, we work with data where the dimension is of the order of~$3$ million, and the number of samples is~$700$ million.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
