发表机构
Technical University of Munich; Technische Universität Berlin; Northern Arizona University（慕尼黑工业大学; 柏林工业大学; 北亚利桑那大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对联邦LoRA中客户端秩异构导致的双线性不匹配问题，提出SeFoRA算法，引入SeFoRA-Ho秩齐次版本，理论保证收敛速率，在GLUE数据集上验证其性能优于现有最优方法。
AI 中文摘要
我们研究大型神经网络的联邦参数高效微调，采用低秩适配（LoRA，Hu等人2022年提出）。将LoRA与联邦参数高效微调（PEFT）结合会产生两者单独不存在的挑战：客户端可能使用不同的LoRA秩，导致其因子矩阵维度不兼容，而逐因子平均会出现双线性不匹配。我们提出SeFoRA，一种草图聚合联邦LoRA算法，每个客户端传输其局部更新的线性草图，支持在聚合器端直接聚合。因此，SeFoRA缓解了双线性不匹配问题，并允许在全模型的小子空间中进行聚合。我们引入名为SeFoRA-Ho的秩齐次版本，该版本允许在此设置下直接聚合适配器。我们证明，对于秩齐次设置，其收敛到一阶平稳点邻域的速率为$\tilde{\bigO}}(1/T)$。在GLUE数据集上微调RoBERTa-Large的数值实验表明，我们的算法优于现有最优方法。
英文摘要
We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.
