掉队者不是灾难：一种带延迟学习的混合联邦学习算法
Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients
摘要：
联邦学习（FL）是一种新的机器学习框架，它在大量分散的计算设备上训练联合模型。现有的方法，例如，联合平均（Federated Averaging，FedAvg）能够通过同步训练联合模型来提供优化保证，但是通常遭受掉队者，即，低计算能力或通信带宽的物联网设备，特别是异构优化问题。为了减轻落伍者的影响，本文提出了一种新的FL算法，即混合联邦学习（HFL），以实现学习效率和效果的平衡。它由两个主要组件组成：同步内核和异步更新器。与传统的同步FL方法不同，我们的HFL引入了异步更新器，该更新器主动从落伍者中提取非同步和延迟的局部权重。提出了一种自适应延迟SGD（AD-SGD）方法，将延迟的局部更新合并到联合模型中。理论分析表明，无论是凸优化问题还是非凸优化问题，该算法的收敛速度都是$\mathcal{O}(\frac{1}{t+τ})$。
英文摘要：
Federated learning (FL) is a new machine learning framework which trains a joint model across a large amount of decentralized computing devices. Existing methods, e.g., Federated Averaging (FedAvg), are able to provide an optimization guarantee by synchronously training the joint model, but usually suffer from stragglers, i.e., IoT devices with low computing power or communication bandwidth, especially on heterogeneous optimization problems. To mitigate the influence of stragglers, this paper presents a novel FL algorithm, namely Hybrid Federated Learning (HFL), to achieve a learning balance in efficiency and effectiveness. It consists of two major components: synchronous kernel and asynchronous updater. Unlike traditional synchronous FL methods, our HFL introduces the asynchronous updater which actively pulls unsynchronized and delayed local weights from stragglers. An adaptive approximation method, Adaptive Delayed-SGD (AD-SGD), is proposed to merge the delayed local updates into the joint model. The theoretical analysis of HFL shows that the convergence rate of the proposed algorithm is $\mathcal{O}(\frac{1}{t+τ})$ for both convex and non-convex optimization problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
