arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.01956math.OC
收藏

分布式二阶优化的最优收缩

Optimal Shrinkage for Distributed Second-Order Optimization

Fangzhao Zhang, Mert Pilanci

AI总结:

针对分布式二阶优化中Hessian逆偏差问题,提出渐近无偏的收缩估计器并用于牛顿步偏差校正,显著提升收敛速率。

摘要:

在这项工作中,我们解决了分布式二阶优化算法中的Hessian逆偏差问题。我们提出了一种基于收缩的Gram矩阵预解式估计器,该估计器渐近无偏,并刻画了其在各向同性情况下的非渐近收敛速率。我们将此估计器应用于分布式二阶优化算法中牛顿步的偏差校正,以及基于随机草图的方法。我们使用解析表达式检验了基于朴素平均的分布式牛顿方法中存在的偏差,并将其与我们提出的无偏方法进行对比。通过在实际和合成数据集上的实验表明,与标准基线和近期提出的方法相比,我们的方法在收敛速率上取得了显著改进。

英文摘要:

In this work, we address the problem of Hessian inversion bias in distributed second-order optimization algorithms. We introduce a novel shrinkage-based estimator for the resolvent of gram matrices which is asymptotically unbiased, and characterize its non-asymptotic convergence rate in the isotropic case. We apply this estimator to bias correction of Newton steps in distributed second-order optimization algorithms, as well as randomized sketching based methods. We examine the bias present in the naive averaging-based distributed Newton's method using analytical expressions and contrast it with our proposed bias-free approach. Our approach leads to significant improvements in convergence rate compared to standard baselines and recent proposals, as shown through experiments on both real and synthetic datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑