Decentralized federated learning (DFL), a serverless variant of federated learning, poses unique challenges for parameter-efficient fine-tuning due to the factorized structure of low-rank adaptation (LoRA). Unlike linear parameters, decentralized aggregation of LoRA updates introduces topology-dependent cross terms that can destabilize training under dynamic communication graphs. We propose \texttt{TAD-LoRA}, a Topology-Aware Decentralized Low-Rank Adaptation framework that coordinates the updates and mixing of LoRA factors to control inter-client misalignment. We theoretically prove the convergence of \texttt{TAD-LoRA} under non-convex objectives, explicitly characterizing the trade-off between topology-induced cross-term error and block-coordinate representation bias governed by the switching interval of alternative training. Experiments under various communication conditions validate our analysis, showing that \texttt{TAD-LoRA} achieves robust performance across different communication scenarios, remaining competitive in strongly connected topologies and delivering clear gains under moderately and weakly connected topologies, with particularly strong results on the MNLI dataset.
RW-LoRA: Communication-Efficient Decentralized LoRA Fine-Tuning via Random Walks
Singapore University of Technology and Design(新加坡科技设计大学);LinkedIn(领英公司);Technical University of Munich(慕尼黑工业大学);Texas A&M University(德克萨斯农工大学);Rutgers University(罗格斯大学)
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China(软件研究所智能软件研究中心,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning
Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications(网络系统架构与融合北京市重点实验室,北京邮电大学);Information Systems Technology and Design Pillar, Singapore University of Technology and Design(新加坡科技设计大学信息系统技术与设计系)
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(胡志明市国家大学科学大学);Vietnam National University, Ho Chi Minh City(胡志明市越南国家大学);Trivita AI
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Beijing Laboratory of Advanced Information Network(北京先进信息网络实验室);Beijing Key Laboratory of Network System Architecture and Convergence(北京网络系统架构与融合重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Department of Electrical and Computer Engineering and Institute for Data Science and Computing(电气与计算机工程系和数据科学与计算研究所);University of Miami(迈阿密大学);Information Systems Technology and Design Pillar(信息系统技术与设计支柱)
LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University(文化和旅游部可持续旅游智能评估技术重点实验室);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China(教育部机器智能与先进计算重点实验室)
Energy-Efficient UAV-assisted LoRa Gateways: A Multi-Agent Optimization Approach
College of Computing, Mohammed VI Polytechnic University (UM6P)(穆莱·易斯六世理工学院计算机学院);Department of Computer Science, Khalifa University(哈利法大学计算机科学系);Concordia Institute for Information Systems Engineering, Concordia University(康科迪亚大学信息系统工程研究所)
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
College of Intelligence and Computing, Tianjin University, Tianjin, China(智能与计算学院,天津大学,天津,中国);College of Computing and Data Science, Nanyang Technological University, Singapore(计算与数据科学学院,南洋理工大学,新加坡)
Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning
School of Mathematics and Statistics, Central South University, China(数学与统计学学院,中南大学);School of Computer Science, The University of Sydney, Australia(计算机科学学院,悉尼大学);Department of Engineering Science, University of Oxford, England(工程科学系,牛津大学)
Decentralized Rank Scheduling for Energy-Constrained Multi-Task Federated Fine-Tuning in Edge-Assisted IoV Networks
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Cyber Science and Technology, Sun Yat-sen University(中山大学网络安全科学与技术学院)
Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
HiThink Research(海思思考研究院);Auburn University(奥本大学);Kent State University(肯特州立大学);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校);Inria(法国国家信息与自动化研究所);University of Montpellier(蒙彼利埃大学);CNRS(法国国家科学研究中心);LNCC(巴西国家计算科学实验室);Fudan University(复旦大学);BEDI Cloud(北电云)
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Imperial College London(帝国理工学院);Technische Universität Berlin (TU Berlin)(柏林工业大学);Ocean University of China(中国海洋大学);The Hong Kong Polytechnic University(香港理工大学)