arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.15216stat.MLcs.DCcs.LGmath.OC
收藏

FedAvg和FedProx的非参数观点:超越驻点

A Non-parametric View of FedAvg and FedProx: Beyond Stationary Points

Northeastern University(东北大学);Duke University(杜克大学);Tsinghua University(清华大学)

Lili Su, Jiaming Xu, Pengkun Yang

摘要:

联邦学习(FL)是一种很有前途的去中心化学习框架,在隐私保护和降低云计算负载方面具有很大的潜力。最近的工作表明,FedAvg和FedProx -两个广泛采用的FL算法-未能达到稳定点的全局优化目标,即使是齐次线性回归问题。此外,它还担心,在存在异质性的情况下,所学到的共同模型可能根本不能很好地推广到当地。在本文中,我们分析了FedAvg和FedProx的收敛性和统计效率,解决了上述两个问题。我们的分析是基于再生核希尔伯特空间(RKHS)中的标准非参数回归,并允许异构的本地数据分布和不平衡的本地数据集。我们证明了估计误差,测量的经验规范或RKHS规范,衰减率为1/t的一般和指数的有限秩内核。在某些异构设置中,这些上限也意味着FedAvg和FedProx都达到了最佳错误率。为了进一步分析量化在每个客户端的异构性的影响,我们提出了一个新的概念,联邦增益的特点,定义为减少估计误差的客户端加入FL。我们发现,当数据的异构性是温和的,一个客户端与有限的本地数据可以受益于一个共同的模型与一个大的联邦增益。数值实验进一步证实了我们的理论研究结果。

英文摘要:

Federated Learning (FL) is a promising decentralized learning framework and has great potentials in privacy preservation and in lowering the computation load at the cloud. Recent work showed that FedAvg and FedProx - the two widely-adopted FL algorithms - fail to reach the stationary points of the global optimization objective even for homogeneous linear regression problems. Further, it is concerned that the common model learned might not generalize well locally at all in the presence of heterogeneity. In this paper, we analyze the convergence and statistical efficiency of FedAvg and FedProx, addressing the above two concerns. Our analysis is based on the standard non-parametric regression in a reproducing kernel Hilbert space (RKHS), and allows for heterogeneous local data distributions and unbalanced local datasets. We prove that the estimation errors, measured in either the empirical norm or the RKHS norm, decay with a rate of 1/t in general and exponentially for finite-rank kernels. In certain heterogeneous settings, these upper bounds also imply that both FedAvg and FedProx achieve the optimal error rate. To further analytically quantify the impact of the heterogeneity at each client, we propose and characterize a novel notion-federation gain, defined as the reduction of the estimation error for a client to join the FL. We discover that when the data heterogeneity is moderate, a client with limited local data can benefit from a common model with a large federation gain. Numerical experiments further corroborate our theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑