arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.04164cs.DCcs.AIcs.CR
收藏

面向差分隐私的高效无线联邦学习的梯度稀疏化

Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy

Kang Wei, Jun Li, Chuan Ma, Ming Ding, Feng Shu, Haitao Zhao, Wen Chen, Hongbo Zhu

AI总结:

针对无线联邦学习中的隐私泄露与通信瓶颈,提出基于梯度稀疏化的框架,在满足差分隐私约束下降低传输开销并保证收敛,实验验证其优于多种基线。

摘要:

联邦学习(FL)使分布式客户端能够在彼此不共享原始数据的情况下协作训练机器学习模型。然而,它存在因上传模型而导致的私有信息泄露问题。此外,随着模型规模的增大,由于有限的传输带宽,训练延迟增加,并且在使用差分隐私(DP)保护时模型性能会下降。在本文中,我们提出了一种在无线信道上基于梯度稀疏化的联邦学习框架,以提高训练效率而不牺牲收敛性能。具体而言,我们首先设计了一种随机稀疏化算法,以保留每个客户端本地训练中一部分梯度元素,从而减轻由DP引起的性能下降并减少无线信道上传输参数的数量。然后,我们通过建模一个非凸联邦学习问题,分析了所提出算法的收敛界。接下来,我们制定了一个时间序列随机优化问题,以在发射功率、平均传输延迟以及客户端的DP要求的约束下,最小化所推导的收敛界。利用李雅普诺夫漂移加惩罚框架,我们为该优化问题开发了一种解析解。我们在三个真实数据集上进行了大量实验,以证明我们提出的算法的有效性。我们表明,我们提出的算法能够充分利用通信与计算之间的交互作用,以优于基线算法,即随机调度、轮询和延迟最小化算法。

英文摘要:

Federated learning (FL) enables distributed clients to collaboratively train a machine learning model without sharing raw data with each other. However, it suffers the leakage of private information from uploading models. In addition, as the model size grows, the training latency increases due to limited transmission bandwidth and the model performance degrades while using differential privacy (DP) protection. In this paper, we propose a gradient sparsification empowered FL framework over wireless channels, in order to improve training efficiency without sacrificing convergence performance. Specifically, we first design a random sparsification algorithm to retain a fraction of the gradient elements in each client's local training, thereby mitigating the performance degradation induced by DP and and reducing the number of transmission parameters over wireless channels. Then, we analyze the convergence bound of the proposed algorithm, by modeling a non-convex FL problem. Next, we formulate a time-sequential stochastic optimization problem for minimizing the developed convergence bound, under the constraints of transmit power, the average transmitting delay, as well as the client's DP requirement. Utilizing the Lyapunov drift-plus-penalty framework, we develop an analytical solution to the optimization problem. Extensive experiments have been implemented on three real life datasets to demonstrate the effectiveness of our proposed algorithm. We show that our proposed algorithms can fully exploit the interworking between communication and computation to outperform the baselines, i.e., random scheduling, round robin and delay-minimization algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑