再论集中式网络优化的原始-对偶乘子方法
Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks
摘要：
原始对偶乘数法（PDMM）最初是为解决一般网络上的可分解优化问题而设计的。在本文中，我们重新PDMM优化集中式网络。我们首先注意到最近提出的方法FedSplit [1]实现了集中式网络的PDMM。在[1]中，不精确的FedSplit（即，基于梯度的FedSplit）也在经验和理论上进行了研究。我们确定了报告的不精确FedSplit性能不佳的原因，这是由于客户端梯度操作的初始化不正确。为了解决不精确的FedSplit问题，我们提出了两个版本的不精确的PDMM，这被称为基于梯度的PDMM（GPDMM）和加速的GPDMM（AGPDMM），分别。与GPDMM相比，AGPDMM加速了GPDMM，但每次迭代从服务器向每个客户端传输的参数数量是GPDMM的两倍。对于一类凸优化问题，我们给出了GPDMM的一个新的收敛界。我们的新的界限是更严格的比不精确FedSplit。我们还研究了AGPDMM和SCAFFOLD的更新表达式，以找到它们的相似之处。研究发现，当客户端每次迭代的梯度步数K=1时，AGPDMM和SCAFFOLD都可以通过适当的参数设置，简化为香草梯度下降.实验结果表明，当K>1时，AGPDMM比SCAFFOLD收敛得快，而GPDMM比SCAFFOLD收敛得稍差。
英文摘要：
The primal-dual method of multipliers (PDMM) was originally designed for solving a decomposable optimisation problem over a general network. In this paper, we revisit PDMM for optimisation over a centralized network. We first note that the recently proposed method FedSplit [1] implements PDMM for a centralized network. In [1], Inexact FedSplit (i.e., gradient based FedSplit) was also studied both empirically and theoretically. We identify the cause for the poor reported performance of Inexact FedSplit, which is due to the improper initialisation in the gradient operations at the client side. To fix the issue of Inexact FedSplit, we propose two versions of Inexact PDMM, which are referred to as gradient-based PDMM (GPDMM) and accelerated GPDMM (AGPDMM), respectively. AGPDMM accelerates GPDMM at the cost of transmitting two times the number of parameters from the server to each client per iteration compared to GPDMM. We provide a new convergence bound for GPDMM for a class of convex optimisation problems. Our new bounds are tighter than those derived for Inexact FedSplit. We also investigate the update expressions of AGPDMM and SCAFFOLD to find their similarities. It is found that when the number K of gradient steps at the client side per iteration is K=1, both AGPDMM and SCAFFOLD reduce to vanilla gradient descent with proper parameter setup. Experimental results indicate that AGPDMM converges faster than SCAFFOLD when K>1 while GPDMM converges slightly worse than SCAFFOLD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
