当联邦学习遇上预训练语言模型的参数高效微调方法
When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods
AI总结：
本文将参数高效微调方法引入联邦学习，系统评估其在不同数据异构性、数据规模和FL场景下的表现，并提出FedPETuning框架以降低通信与本地适配成本。
摘要：
随着数据隐私问题日益受到关注，近期研究在隐私敏感的自然语言处理（NLP）任务上使用联邦学习（FL）取得了显著进展。大量文献表明，在FL范式中对预训练语言模型（PLM）进行全量微调，可以缓解数据异构性问题，并缩小与集中式训练之间的性能差距。然而，大型PLM给FL系统带来了难以承受的通信开销和本地模型适配成本。为此，我们将多种参数高效微调（PETuning）方法引入联邦学习。具体而言，我们对FL中具有代表性的PLM微调方法进行了全面的实证研究。实验结果涵盖了对数据异构性水平、数据规模以及不同FL场景的分析。通过在本地微调并在全局聚合轻量级模型参数，总体通信开销可以显著降低，同时在各种FL设置下保持可接受的性能。为促进FL中PETuning的研究，我们还开发了一个联邦微调框架FedPETuning，使从业者能够方便地在FL训练范式下利用不同的PETuning方法。源代码可在 https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning 获取。
英文摘要：
With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
