arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.09086cs.LG
收藏

联邦学习中的掉队客户端学习

Learning from straggler clients in federated learning

Google Research(谷歌研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Google DeepMind(谷歌DeepMind)

Andrew Hard, Antonious M. Girgis, Ehsan Amid, Sean Augenstein, Lara McConnaughey, Rajiv Mathews, Rohan Anil

AI总结:

本文针对联邦学习中延迟严重的掉队客户端,提出基于蒸馏和平均的两种新算法FARe-DUST与FeAST-on-MSG,在多个基准任务上提升了准确性并优化了训练时间与精度的权衡。

摘要:

现有的联邦学习算法在从返回模型更新存在显著时间延迟的客户端设备中学习时表现如何?是否有可能从在调度后数分钟、数小时或数天后才报告回传的客户端中有效学习?我们通过开发由实际应用引导的客户端延迟蒙特卡洛模拟来回答这些问题。我们研究了同步优化算法如FedAvg和FedAdam,以及异步的FedBuff算法,并观察到所有这些现有方法都难以从严重延迟的客户端中学习。为了改善这一状况,我们尝试了多种修改,包括蒸馏正则化和模型权重的指数移动平均。最后,我们引入了两种新算法,FARe-DUST和FeAST-on-MSG,分别基于蒸馏和平均。在EMNIST、CIFAR-100和StackOverflow基准联邦学习任务上的实验表明,我们的新算法在掉队客户端的准确性方面优于现有算法,同时在训练时间和总体准确性之间提供了更好的权衡。

英文摘要:

How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay? Is it even possible to learn effectively from clients that report back minutes, hours, or days after being scheduled? We answer these questions by developing Monte Carlo simulations of client latency that are guided by real-world applications. We study synchronous optimization algorithms like FedAvg and FedAdam as well as the asynchronous FedBuff algorithm, and observe that all these existing approaches struggle to learn from severely delayed clients. To improve upon this situation, we experiment with modifications, including distillation regularization and exponential moving averages of model weights. Finally, we introduce two new algorithms, FARe-DUST and FeAST-on-MSG, based on distillation and averaging, respectively. Experiments with the EMNIST, CIFAR-100, and StackOverflow benchmark federated learning tasks demonstrate that our new algorithms outperform existing ones in terms of accuracy for straggler clients, while also providing better trade-offs between training time and total accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑