服务器端步长和无替换采样可证明有助于联邦优化
Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization
摘要：
本文对联邦学习中的服务器端优化进行了理论研究。我们的研究结果是第一个表明，广泛流行的启发式扩展客户端更新与一个额外的参数是非常有用的背景下，联邦平均（FedAvg）与本地通过客户端数据。每次本地传递都使用随机重排执行，无需替换，这是我们可以显示出更高复杂性的关键原因。特别是，我们证明，每当局部步长很小，更新方向是由FedAvg与随机重排在所有客户端，可以采取一个大的飞跃，在获得的方向和提高率凸，强凸，非凸目标。特别地，在非凸区域中，我们得到了从$\mathcal{O}\left(\varepsilon^{-3}\right)$到$\mathcal{O}\left(\varepsilon^{-2}\right)$的收敛速度的增强。这个结果甚至对于在单个节点上执行的随机重排也是新的。相比之下，如果本地步长较大，我们证明了客户端采样的噪声可以通过使用一个小的服务器端步长来控制。据我们所知，这是第一次证明地方步骤有助于克服沟通瓶颈。总之，我们的研究结果的优势，大，小的服务器端步长提供了一个正式的理由的做法，自适应服务器端优化联邦学习。此外，我们认为我们的算法的一个变种，支持部分客户端的参与，这使得该方法更实用。
英文摘要：
We present a theoretical study of server-side optimization in federated learning. Our results are the first to show that the widely popular heuristic of scaling the client updates with an extra parameter is very useful in the context of Federated Averaging (FedAvg) with local passes over the client data. Each local pass is performed without replacement using Random Reshuffling, which is a key reason we can show improved complexities. In particular, we prove that whenever the local stepsizes are small, and the update direction is given by FedAvg in conjunction with Random Reshuffling over all clients, one can take a big leap in the obtained direction and improve rates for convex, strongly convex, and non-convex objectives. In particular, in non-convex regime we get an enhancement of the rate of convergence from $\mathcal{O}\left(\varepsilon^{-3}\right)$ to $\mathcal{O}\left(\varepsilon^{-2}\right)$. This result is new even for Random Reshuffling performed on a single node. In contrast, if the local stepsizes are large, we prove that the noise of client sampling can be controlled by using a small server-side stepsize. To the best of our knowledge, this is the first time that local steps provably help to overcome the communication bottleneck. Together, our results on the advantage of large and small server-side stepsizes give a formal justification for the practice of adaptive server-side optimization in federated learning. Moreover, we consider a variant of our algorithm that supports partial client participation, which makes the method more practical.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
