基于有损分布式信源编码的联邦学习:分析与优化
Federated Learning with Lossy Distributed Source Coding: Analysis and Optimization
摘要:
最近,联邦学习(FL)取代了数据共享与模型共享,已成为一种高效和隐私友好的机器学习(ML)范式。FL的主要挑战之一是模型聚合的巨大通信成本。已经提出了许多压缩/量化方案来减少模型聚合的通信成本。然而,以下问题仍然没有答案:什么是基本的通信成本和FL收敛性能之间的权衡?在本文中,我们设法回答这个问题。具体而言,我们首先提出了一个通用的框架模型聚合性能分析的基础上率失真理论。在所提出的分析框架下,我们推导出模型聚合率失真区域的内界。然后,我们进行FL收敛分析连接聚合失真和FL收敛性能。我们制定了一个聚合失真最小化问题,以提高FL收敛性能。两个算法被开发来解决上述问题。数值结果表明,聚合失真,收敛性能和通信成本的基线模型聚合方案仍然有很大的潜力,进一步改进。
英文摘要:
Recently, federated learning (FL), which replaces data sharing with model sharing, has emerged as an efficient and privacy-friendly machine learning (ML) paradigm. One of the main challenges in FL is the huge communication cost for model aggregation. Many compression/quantization schemes have been proposed to reduce the communication cost for model aggregation. However, the following question remains unanswered: What is the fundamental trade-off between the communication cost and the FL convergence performance? In this paper, we manage to answer this question. Specifically, we first put forth a general framework for model aggregation performance analysis based on the rate-distortion theory. Under the proposed analysis framework, we derive an inner bound of the rate-distortion region of model aggregation. We then conduct an FL convergence analysis to connect the aggregation distortion and the FL convergence performance. We formulate an aggregation distortion minimization problem to improve the FL convergence performance. Two algorithms are developed to solve the above problem. Numerical results on aggregation distortion, convergence performance, and communication cost demonstrate that the baseline model aggregation schemes still have great potential for further improvement.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。