arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00451cs.LGcs.DC
收藏

通过拓扑感知交替LoRA稳定去中心化联邦微调

Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRA

Department of Electrical and Computer Engineering, New York University, Brooklyn, NY, USA(电气与计算机工程系,纽约大学,布鲁克林,纽约,美国)

Xiaoyu Wang, Xiaotian Li, Zhixiang Zhou, Chen Li, Yong Liu

AI总结:

TAD-LoRA通过拓扑感知的交替LoRA方法,在去中心化联邦学习中稳定参数高效微调,提升不同通信拓扑下的性能。

摘要:

去中心化联邦学习(DFL)作为联邦学习的一种无服务器变种,在参数高效微调方面面临独特挑战,这归因于低秩适应(LoRA)的因子化结构。与线性参数不同,去中心化聚合LoRA更新引入了拓扑依赖的交叉项,这些项在动态通信图下可能破坏训练稳定性。我们提出TAD-LoRA,一种拓扑感知的去中心化低秩适应框架,该框架协调LoRA因素的更新和混合以控制客户端间的不一致。我们理论证明了TAD-LoRA在非凸目标下的收敛性,明确刻画了由替代训练切换间隔决定的拓扑诱导交叉项误差与块坐标表示偏差之间的权衡。实验在各种通信条件下验证了我们的分析,显示TAD-LoRA在不同通信场景中均表现出鲁棒性,保持在强连接拓扑中的竞争力,并在中等和弱连接拓扑中实现明显优势,尤其在MNLI数据集上表现特别强劲。

英文摘要:

Decentralized federated learning (DFL), a serverless variant of federated learning, poses unique challenges for parameter-efficient fine-tuning due to the factorized structure of low-rank adaptation (LoRA). Unlike linear parameters, decentralized aggregation of LoRA updates introduces topology-dependent cross terms that can destabilize training under dynamic communication graphs. We propose \texttt{TAD-LoRA}, a Topology-Aware Decentralized Low-Rank Adaptation framework that coordinates the updates and mixing of LoRA factors to control inter-client misalignment. We theoretically prove the convergence of \texttt{TAD-LoRA} under non-convex objectives, explicitly characterizing the trade-off between topology-induced cross-term error and block-coordinate representation bias governed by the switching interval of alternative training. Experiments under various communication conditions validate our analysis, showing that \texttt{TAD-LoRA} achieves robust performance across different communication scenarios, remaining competitive in strongly connected topologies and delivering clear gains under moderately and weakly connected topologies, with particularly strong results on the MNLI dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑