arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.03863cs.LG
收藏

我们能否从理论上量化局部更新对联邦学习泛化性能的影响?

Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?

The Ohio State University(俄亥俄州立大学);Rochester Institute of Technology(罗切斯特理工大学)

Peizhong Ju, Haibo Yang, Jia Liu, Yingbin Liang, Ness Shroff

AI总结:

本文针对数据异质性和局部更新如何影响联邦学习泛化的问题,基于线性模型给出模型误差闭式解,刻画局部更新次数、轮数及参数配置的作用。

摘要:

联邦学习(Federated Learning,FL)因能够在无需直接共享数据的情况下跨多个不同站点训练机器学习模型而受到广泛欢迎。尽管各种算法及其优化分析已经表明,采用局部更新的 FL 是一种通信高效的分布式学习框架,但采用局部更新的 FL 的泛化性能受到的关注相对较少。这种研究不足可归因于 FL 框架内数据异质性与由局部更新导致的不频繁通信之间复杂的相互作用。这促使我们研究 FL 中的一个基础性问题:随着学习过程的推进,我们能否量化数据异质性和局部更新对 FL 泛化性能的影响?为此,作为第一步,我们使用线性模型对 FL 的泛化性能进行全面的理论研究,其中在静态情形以及在线/非静态情形下均考虑了数据异质性。通过给出模型误差的闭式表达式,我们严格量化了三种设置($K=1$、$K<\infty$ 和 $K=\infty$)下局部更新次数(记为 $K$)的影响,并展示泛化性能如何随通信轮数 $t$ 演化。我们的研究还全面理解了不同配置(包括模型参数数量 $p$ 和训练样本数量 $n$)如何影响整体泛化性能,从而为通过网络实施 FL 提供了新的见解(例如良性过拟合)。

英文摘要:

Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention. This lack of investigation can be attributed to the complex interplay between data heterogeneity and infrequent communication due to the local updates within the FL framework. This motivates us to investigate a fundamental question in FL: Can we quantify the impact of data heterogeneity and local updates on the generalization performance for FL as the learning process evolves? To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases. By providing closed-form expressions of the model error, we rigorously quantify the impact of the number of the local updates (denoted as $K$) under three settings ($K=1$, $K<\infty$, and $K=\infty$) and show how the generalization performance evolves with the number of rounds $t$. Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters $p$ and the number of training samples $n$) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for implementing FL over networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑