arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01024cs.DC
收藏

面向无线网络的低延迟联邦微调框架用于大语言模型

Low-Latency Federated Fine-Tuning for Large Language Models Over Wireless Networks

Nanjing University of Science and Technology(南京理工大学);Southeast University(东南大学);Hainan University(海南大学)

Zhiwen Pang, Kang Wei, Long Shi, Zhe Wang, Jun Li, Feng Shu

AI总结:

本文提出了一种联合客户端特定剪枝和带宽分配框架,以提高无线网络下大语言模型的微调效率,实验表明其在减少训练时间的同时降低了计算和通信开销。

摘要:

近年来,由于大语言模型(LLM)与联邦学习(FL)相结合的能力,联邦大语言模型受到了广泛关注,这种能力能够解决协作微调中的隐私问题。然而,由于LLM参数规模大,现有的联邦LLM微调框架在资源受限的客户端上面临挑战,这些客户端具有异构的计算能力和随机的无线信道。为了解决这个问题,我们提出了一种联合客户端特定剪枝和带宽分配(JCPBA)框架,用于联邦LLM,以提高在无线网络上的微调效率。具体来说,我们通过联合优化剪枝率和带宽分配来制定一个微调延迟最小化问题。此外,我们使用块坐标下降法来解决这个问题。在Yahoo Answers和GSM8K数据集上的广泛实验表明,所提出的框架在与现有最先进基线相比时,显著减少了实时时钟微调时间,并在计算和通信开销较低的情况下获得了相等或更低的测试损失。

英文摘要:

Recently, federated large language models (LLMs) have drawn significant attention thanks to coupled capabilities of LLMs and federated learning (FL) that address privacy concerns in collaborative fine-tuning. However, due to large-scale parameters of LLMs, existing federated LLM fine-tuning frameworks incur significant challenges in resource-constrained clients characterized by heterogeneous computing capabilities and random wireless channels. To address this issue, we propose a joint client-specific pruning and bandwidth allocation (JCPBA) framework for federated LLMs to improve the fine-tuning efficiency over the wireless networks. Specifically, we formulate a fine-tuning latency minimization problem by jointly optimizing pruning rates and bandwidth allocations. Furthermore, we solve this optimization problem using a block coordinate descent method. Extensive experiments on the datasets of Yahoo Answers and GSM8K demonstrate that the proposed framework significantly reduces wall-clock fine-tuning time compared with state-of-the-art baselines and gains equal or lower test loss at the cost of lower computation and communication overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑