arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.00902cs.DC
收藏

弹性秩专家混合用于联邦微调中的知识重用

Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning

University of Macau(澳门大学);HKUST(香港科技大学);HKUST (Guangzhou)(香港科技大学(广州))

Yebo Wu, Jingguang Li, Zhijiang Guo, Li Li

AI总结:

SmartFed通过弹性秩专家混合和弹性专家配额分配,实现了高效的联邦微调,有效重用已有知识以提升模型性能和训练效率。

摘要:

联邦微调为适应大型语言模型(LLMs)到下游任务提供了有前途的解决方案,同时保护数据隐私。然而,其高计算和通信需求阻碍了其在资源受限设备上的部署。在本文中,我们提出了SmartFed,一个资源高效的联邦微调框架。SmartFed智能地重用现有LoRA模块中嵌入的知识,在适应LLMs到新任务时无需进行昂贵的从头训练。为了有效利用这种知识并确保可扩展性,我们引入了秩专家混合(MoRE)。MoRE将LoRA模块分解为细粒度的秩级专家。这些专家根据输入语义和资源预算被选择性激活并组合。此外,为了优化资源利用率,我们提出了弹性专家配额分配(EEQA)。EEQA根据其对模型性能的贡献,在参数矩阵之间自适应地分配专家容量,将计算资源集中在关键专家上。在多个基准上的广泛评估表明,SmartFed在模型性能和训练效率上显著优于现有方法。

英文摘要:

Federated fine-tuning offers a promising solution for adapting Large Language Models (LLMs) to downstream tasks while safeguarding data privacy. However, its high computational and communication demands hinder its deployment on resource-constrained devices. In this paper, we propose SmartFed, a resource-efficient federated fine-tuning framework. SmartFed intelligently reuses knowledge embedded in existing LoRA modules, eliminating the need for expensive training from scratch when adapting LLMs to new tasks. To effectively exploit this knowledge and ensure scalability, we introduce the Mixture of Rank-Wise Experts (MoRE). MoRE decomposes LoRA modules into fine-grained rank-level experts. These experts are selectively activated and combined based on input semantics and resource budgets. Moreover, to optimize resource utilization, we present the Elastic Expert Quota Allocation (EEQA). EEQA adaptively allocates expert capacity across parameter matrices based on their contribution to model performance, focusing computing resources on the critical experts. Extensive evaluations across multiple benchmarks demonstrate that SmartFed significantly outperforms existing methods in model performance and training efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑