arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.02001cs.DC
收藏

EcoLoRA:面向大语言模型的通信高效联邦微调

EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models

Washington University in St. Louis(华盛顿大学圣路易斯分校);The Ohio State University(俄亥俄州立大学);Virginia Polytechnic Institute and State University(弗吉尼亚理工大学)

Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang

AI总结:

EcoLoRA通过轮转LoRA分段共享、自适应稀疏化和无损编码,在保持大语言模型联邦微调性能的同时,将通信时间最多降低79%、总训练时间最多降低65%。

摘要:

为解决数据本地化和隐私限制问题,联邦学习(FL)近来已被用于微调大语言模型(LLMs),从而无需聚合数据即可提升各种下游任务的性能。然而,联邦学习中模型更新的反复交换会导致极高的通信成本,阻碍分布式学习过程。为应对这一挑战,我们提出EcoLoRA,一种新颖的面向大语言模型的通信高效联邦微调框架。利用模块化结构,我们提出了一种轮转分段共享方案,其中每个客户端每轮仅上传一个互补的LoRA分段以降低网络带宽。该方案进一步结合了针对LoRA训练动态定制的自适应稀疏化方法和无损编码技术。我们在多个数据集和模型上对问答任务和价值对齐任务进行了广泛评估。结果表明,EcoLoRA在不牺牲性能的情况下显著降低了通信开销。例如,它将通信时间最多减少79%,总训练时间最多减少65%。

英文摘要:

To address data locality and privacy restrictions, Federated Learning (FL) has recently been adopted to fine-tune large language models (LLMs), enabling improved performance on various downstream tasks without requiring aggregated data. However, the repeated exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. To address this challenge, we propose EcoLoRA, a novel communication-efficient federated fine-tuning framework for LLMs. Leveraging the modular structure, we propose a round-robin segment sharing scheme, where each client uploads only a complementary LoRA segment per round to reduce network bandwidth. It is further combined with adaptive sparsification methods tailored to LoRA's training dynamics and lossless encoding techniques. We conduct extensive evaluations on both question-answering and value-alignment tasks across multiple datasets and models. The results show that EcoLoRA significantly reduces communication overhead without compromising performance. For instance, it reduces communication time by up to 79% and total training time by up to 65%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑