发表机构
DGIST; Samsung（大邱庆北科学技术院; 三星）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
FedPA-LoRA是缓解异构联邦LoRA聚合与初始化误差的产品对齐框架，在多任务实验中较基线方法平均GLUE准确率最高提升6.82个百分点。
AI 中文摘要
低秩适配（LoRA）支持大型语言模型的高效联邦微调，但其因子化参数化在本地更新的准确聚合与局部优化因子的连续性之间存在权衡：逐因子聚合会产生聚合不匹配，但能更好保留因子连续性；而乘积空间重构虽减少了这种不匹配，却会因新重构因子导致更严重的因子级初始化不匹配。本文提出FedPA-LoRA，这是一种产品对齐的联邦LoRA框架，可联合解决上述限制，且在客户端秩同质和异构场景下均能保证收敛。每个客户端在通信轮次中保留其局部因子，并将自身乘积向特定秩的全局参考对齐，在数据异质性下维持局部优化连续性的同时促进全局一致性。服务器在公共乘积空间中聚合异构秩的更新，并高效重构秩约束的全局适配器，无需形成密集聚合。该设计支持客户端特定的计算与通信预算。在自然语言理解与生成任务上的实验表明，FedPA-LoRA在不同数据异质性水平及秩同质、异构设置下，均显著优于代表性基线；在客户端秩异构场景下，平均GLUE准确率提升最高达6.82个百分点。
英文摘要
Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.
Comments35 pages, 6 figures. Code: https://github.com/Juseok-Jeon/FedPA-LoRA
