arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.01708cs.LGcs.CV
收藏

探索参数高效微调以在联邦学习中启用基础模型

Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

University of Central Florida(中佛罗里达大学);University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校)

Guangyu Sun, Umar Khalid, Matias Mendieta, Pu Wang, Chen Chen

AI总结:

研究在联邦学习中应用参数高效微调,提出FedPEFT框架,通过仅本地微调并全局共享少量权重,在多种设置下显著降低通信开销并保持优异性能。

摘要:

联邦学习(FL)已成为一种有前景的范式,能够在不集中访问本地设备原始数据的情况下实现模型的协作训练。在典型的FL范式(如FedAvg)中,模型权重每轮在服务器与参与客户端之间来回传输。最近,使用小型预训练模型已被证明在联邦学习优化和提高收敛性方面是有效的。然而,最近最先进的预训练模型能力越来越强,但参数也越来越多,被称为“基础模型”。在传统FL中,共享庞大的模型权重会迅速给系统带来巨大的通信负担,尤其是在采用更强能力的模型时。我们能否找到一种解决方案,在FL中启用这些强大且现成的预训练模型,在实现卓越性能的同时减轻通信负担?为此,我们研究了参数高效微调在联邦学习中的应用,并由此引入了一个新框架:FedPEFT。具体而言,我们系统地评估了FedPEFT在多种客户端稳定性、数据分布和差分隐私设置下的性能。通过仅在本地微调并全局共享一小部分模型权重,可以在广泛的联邦学习场景中实现总通信开销的显著降低,同时保持具有竞争力甚至更好的性能,为实用且有效的联邦系统提供了新范式的见解。

英文摘要:

Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models." In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑