arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.15482cs.LGstat.ML
收藏

基于高斯过程的个性化联邦学习

Personalized Federated Learning with Gaussian Processes

Bar-Ilan University(巴伊兰大学);NVIDIA(英伟达)

Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, Ethan Fetaya

摘要:

联邦学习旨在学习一种全局模型,该模型在跨客户端通信有限的客户端设备上表现良好。个性化联邦学习(PFL)进一步扩展了这种设置,通过学习个性化模型来处理客户端之间的数据异构性。在这种情况下,一个关键的挑战是跨客户端有效地学习,即使每个客户端都有唯一的数据,这些数据通常在大小上是有限的。在这里,我们提出了pFedGP,这是一种基于高斯过程(GP)和深度内核学习的PFL解决方案。GP是高度表达的模型,由于其贝叶斯性质,在低数据状态下运行良好。然而,将GPS应用于PFL提出了多个挑战。GP的性能主要取决于对一个好的内核函数的访问,而学习内核需要一个大的训练集。因此,我们建议在所有客户端之间学习一个共享的内核函数,由神经网络参数化,每个客户端都有一个个人GP分类器。我们进一步扩展pFedGP包括诱导点使用两种新的方法,第一种方法有助于提高低数据制度的泛化,第二种方法降低了计算成本。我们推导出一个PAC-Bayes泛化约束的新客户端和经验表明,它提供了非空的保证。在CIFAR-10、CIFAR-100和CINIC-10的标准PFL基准测试以及输入噪声下的新学习设置上进行的广泛实验表明,pFedGP实现了良好的校准预测,同时显著优于基线方法,准确度增益高达21%。

英文摘要:

Federated learning aims to learn a global model that performs well on client devices with limited cross-client communication. Personalized federated learning (PFL) further extends this setup to handle data heterogeneity between clients by learning personalized models. A key challenge in this setting is to learn effectively across clients even though each client has unique data that is often limited in size. Here we present pFedGP, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning. GPs are highly expressive models that work well in the low data regime due to their Bayesian nature. However, applying GPs to PFL raises multiple challenges. Mainly, GPs performance depends heavily on access to a good kernel function, and learning a kernel requires a large training set. Therefore, we propose learning a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client. We further extend pFedGP to include inducing points using two novel methods, the first helps to improve generalization in the low data regime and the second reduces the computational cost. We derive a PAC-Bayes generalization bound on novel clients and empirically show that it gives non-vacuous guarantees. Extensive experiments on standard PFL benchmarks with CIFAR-10, CIFAR-100, and CINIC-10, and on a new setup of learning under input noise show that pFedGP achieves well-calibrated predictions while significantly outperforming baseline methods, reaching up to 21% in accuracy gain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑