arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.04472cs.LGcs.AIcs.DC
收藏

面向基础模型训练的高效联邦学习方法综述

A Survey on Efficient Federated Learning Methods for Foundation Model Training

Technical University of Munich(慕尼黑工业大学);IBM Research(IBM研究院);University of Bayreuth(拜罗伊特大学);University of Toronto(多伦多大学)

Herbert Woisetschläger, Alexander Isenko, Shiqiang Wang, Ruben Mayer, Hans-Arno Jacobsen

AI总结:

本文综述面向基础模型训练的高效联邦学习方法,围绕计算与通信效率提出分类体系,分析 PEFT 的优劣、框架适配性及生成模型评估与隐私研究方向。

摘要:

联邦学习(Federated Learning,FL)已成为一项成熟技术,用于在众多客户端之间促进隐私保护的协同训练。然而,新的 FL 方法在讨论其贡献时往往只涉及小型深度学习模型,并聚焦于在客户端上训练完整模型。随着基础模型(Foundation Models,FM)的出现,许多深度学习应用面临的现实情况有所不同。通常,FM 已经在种类广泛的任务上完成预训练,并且可以在比完整模型训练所需规模小得多的数据集上,针对特定下游任务进行微调。然而,获取此类数据集往往具有挑战性。按照其设计,FL 有助于打通数据孤岛。通过本综述,我们提出了一种聚焦计算与通信效率的全新分类体系,这些效率要素是在 FL 系统中利用 FM 的关键。我们讨论了面向 FL 应用的参数高效微调(parameter-efficient fine-tuning,PEFT)的优势与缺点,详细阐述了 FL 框架与 FM 协同工作的就绪程度,并给出了未来研究机会,包括如何在 FL 中评估生成式模型,以及隐私与 PEFT 之间的相互作用。

英文摘要:

Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑