发表机构
University of Florida; Middle Tennessee State University（佛罗里达大学; 中田纳西州立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对秩异质性联邦学习中现有方法无法捕捉客户端特定特征的问题，本文提出FedRoRA框架，通过解耦适配方向与幅度实现细粒度个性化，在NLU与NLG基准上性能优于现有最优方法。
AI 中文摘要
大语言模型（LLMs）在诸多领域取得了显著成功，但将其适配到隐私敏感的分布式数据集仍是一项挑战。联邦学习（FL）结合低秩适配（LoRA）为协同微调提供了资源高效的范式，但实际部署受限于资源异质性与数据异质性的双重挑战。现有秩异质性方法主要聚焦于弥合聚合时的维度不匹配，却通常为所有相同秩的客户端提供统一的全局模型，无法捕捉非独立同分布（non-IID）场景下的客户端特定特征。本文提出FedRoRA（Federated Rank-wise Personalized LoRA，联邦秩级个性化LoRA），一种能在秩异质性联邦中实现细粒度个性化的新型框架。FedRoRA将适配解耦为共享全局方向与由可学习对角尺度调控的个性化秩级幅度。在服务器端，它通过奇异值分解（SVD）提取全局子空间，并通过个性化投影与top-k选择机制重新分配客户端特定初始化。在自然语言理解（NLU）与自然语言生成（NLG）基准上的大量实验表明，FedRoRA的性能始终优于现有最优方法。
英文摘要
Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-$k$ selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.
CommentsAccepted to EMNLP 2026
