高效通信的联邦微调
FDA-Opt: Federated Fine-Tuning via Dynamic Update Schedules
AI总结:
本文提出FDA-Opt算法,解决联邦学习中参数通信频繁和刚性的难题,通过统一FDA和FedOpt方法,提升语言模型在下游NLP任务中的微调性能。
摘要:
联邦学习(FL)使利用大量之前无法获取的数据源成为可能。同时,预训练语言模型(LMs)已席卷全球,原因在于它们表现出显著的涌现能力,并能迅速适应下游任务。这为FL开辟了最激动人心的前沿:微调LMs。然而,FL中一个持续的挑战是参数频繁且刚性的通信——这一问题在当前模型规模庞大时更加突出。FedOpt家族的算法已成为FL的首选方法,依赖于固定但任意的间隔进行模型交换。最近,FDA算法提出了一种动态方法,通过监控训练进度。然而,它引入了一个难以校准的参数,并施加了刚性的同步方案。在本文中,我们通过提出FDA-Opt家族的算法——FDA和FedOpt的统一泛化,解决了这些限制。我们的实验评估集中在微调LMs以完成下游NLP任务上,并证明了FDA-Opt在配置为专门优化FedOpt的超参数时仍优于FedOpt。换句话说,我们展示了FDA-Opt是现代FL库和系统中的实用、即插即用的替代品:它不需要额外的配置,并能即开即用地提供更优的性能。
英文摘要:
Federated Learning (FL) enables the utilization of vast, previously inaccessible data sources. At the same time, pre-trained Language Models (LMs) have taken the world by storm and for good reason. They exhibit remarkable emergent abilities and are readily adapted to downstream tasks. This opens one of the most exciting frontiers in FL: fine-tuning LMs. Yet, a persistent challenge in FL is the frequent, rigid communication of parameters -- a problem magnified by the sheer size of these contemporary models. The FedOpt family of algorithms has become the go-to approach for FL, relying on fixed but arbitrary intervals for model exchanges. Recently, the FDA algorithm prescribed a dynamic approach by monitoring the training progress. However, it introduced a hard-to-calibrate parameter and imposed a rigid synchronization scheme. In this work, we address these limitations by proposing the FDA-Opt family of algorithms -- a unified generalization of both FDA and FedOpt. Our experimental evaluation focuses on fine-tuning LMs on downstream NLP tasks and demonstrates that FDA-Opt outperforms FedOpt even when it is configured with hyper-parameters specifically optimized for the latter. In other words, we show that FDA-Opt is a practical, drop-in replacement for FedOpt in modern FL libraries and systems: it requires no additional configuration and delivers superior performance out of the box.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。