Federated learning (FL) learns a model jointly from a set of participating devices without sharing each other's privately held data. The characteristics of non-i.i.d. data across the network, low device participation, high communication costs, and the mandate that data remain private bring challenges in understanding the convergence of FL algorithms, particularly regarding how convergence scales with the number of participating devices. In this paper, we focus on Federated Averaging (FedAvg), one of the most popular and effective FL algorithms in use today, as well as its Nesterov accelerated variant, and conduct a systematic study of how their convergence scale with the number of participating devices under non-i.i.d. data and partial participation in convex settings. We provide a unified analysis that establishes convergence guarantees for FedAvg under strongly convex, convex, and overparameterized strongly convex problems. We show that FedAvg enjoys linear speedup in each case, although with different convergence rates and communication efficiencies. For strongly convex and convex problems, we also characterize the corresponding convergence rates for the Nesterov accelerated FedAvg algorithm, which are the first linear speedup guarantees for momentum variants of FedAvg in convex settings. Empirical studies of the algorithms in various settings have supported our theoretical results.
Katyusha: The First Direct Acceleration of Stochastic Gradient Methods
Princeton University（普林斯顿大学）；Institute for Advanced Study（高等研究院）
Nesterov的动量技巧以加速梯度下降而闻名，并已被证明在构建快速迭代算法方面很有用。然而，在随机设置中，存在反例，并阻止Nesterov的动量提供类似的加速度，即使潜在的问题是凸和有限和。我们引入$\mathtt{Katyusha}$，一个直接的，原始的唯一的随机梯度法来解决这个问题。在凸有限和随机最优化中，$\mathtt{Katyusha}$具有最优加速收敛速度，并且在小批量环境下具有最优并行线性加速比。主要成分是$\textit{Katyusha momentum}$，一种新的“负动力”，位于涅斯捷罗夫的动力之上。它可以被合并到一个方差减少为基础的算法，并加快它，无论是在$\textit{sequential and parallel}$性能方面。由于方差缩减已成功地应用于越来越多的实际问题，我们的论文建议，在每一个这样的情况下，人们可能会尝试给喀秋莎一个拥抱。
Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis
Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Shenzhen University（深圳大学）；University of Science and Technology Beijing（北京科技大学）；Carleton University（卡尔顿大学）；The University of British Columbia（不列颠哥伦比亚大学）
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong Baptist University（香港浸会大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
