Large Language Models (LLMs) are pivotal in natural language processing. The impracticality of full fine-tuning has prompted Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA), optimizing low-rank matrices A and B. In distributed scenarios where privacy constraints necessitate Federated Learning (FL), however, the integration of LoRA is often unstable. Specifically, we identify that aggregating updates from multiple clients introduces statistical variance that scales with the client count, causing gradient collapse when using high-rank adapters. Existing scaling factor candidates, such as the one used by Rank-Stabilized LoRA, ignore the interaction caused by the aggregation process. To bridge this gap, this paper introduces Stabilized Federated LoRA (SFed-LoRA), a framework that theoretically characterizes the interaction between adapter rank and federated aggregation. We derive an optimal scaling factor designed to effectively mitigate the aggregation error accumulating across N clients. By correcting the scaling mismatch inherent in previous approaches, SFed-LoRA restores the efficacy of high-rank adaptation without altering the original model architecture or increasing inference latency. Extensive experiments in diverse tasks, model architectures, and heterogeneous data distributions are conducted to validate our results. We demonstrate that SFed-LoRA prevents high-rank collapse, and achieves significantly improved stability and faster convergence compared with state-of-the-art baselines for high-rank adaptation.
$\textbf{AGT$^{AO}$}$: Robust and Stabilized LLM Unlearning via Adversarial Gating Training with Adaptive Orthogonality
School of Computer Science and Technology, Xi’an Jiaotong University, China（西安交通大学计算机科学与技术学院）；MOE KLINNS Lab, Xi’an Jiaotong University, China（西安交通大学MOE KLINNS实验室）；Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China（陕西省大数据知识工程重点实验室）
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；ByteDance Inc.（字节跳动公司）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Southeast University, Nanjing, China（南京东南大学）；Wuhan AI Research, Wuhan, China（武汉人工智能研究院）
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China（软件研究所智能软件研究中心，中国科学院，北京）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京）
Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction
School of Software Engineering, South China University of Technology, Guangzhou, China（华南理工大学软件学院）；Key Laboratory of Big Data and Intelligent Robot (SCUT), MOE of China（大数据与智能机器人重点实验室）；Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学计算机系）；School of Electrical Engineering, Guangxi University, Nanning, China（广西大学电气工程学院）；Department of Biostatistics & Health Informatics, King’s College London, London, United Kingdom（伦敦国王学院生物统计与健康信息学系）
One-for-All: A Lightweight Stabilized and Parameter-Efficient Pre-trained LLM for Time Series Forecasting
ADAPT SFI Research Centre, School of Computer Science, Technological University Dublin（ADAPT SFI研究中心，计算机科学学院，都柏林科技大学）；ADAPT SFI Research Centre, School of Computer Science, University College Dublin（ADAPT SFI研究中心，计算机科学学院，都柏林大学学院）
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University（自动化系，清华大学）；Academy of Medical Engineering and Translational Medicine, Tianjin University（医学工程与转化医学学院，天津大学）
FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning
School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）；Department of Electronic and Electrical Engineering, Southern University of Science and Technology（电子与电气工程系，南方科技大学）；School of Electronic Information, Wuhan University（电子信息学院，武汉大学）；Centre for Wireless Communications, University of Oulu（无线通信中心，奥卢大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气与电子工程系，香港大学）
RAIE: Region-Aware Incremental Preference Editing with LoRA for LLM-based Recommendation
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）；Department of Computer Science and Technology, Xiamen University（厦门大学计算机科学与技术系）；Artificial Intelligence Research Institute, Shenzhen MSU-BIT University（深圳MSU-BIT大学人工智能研究院）；School of Information, Central University of Finance and Economics（中央财经大学信息学院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
Low-Rank Adaptation for Foundation Models: A Comprehensive Review
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Yale University（耶鲁大学）；Nanyang Technological University（南洋理工大学）；Central University of Finance and Economics（中央财经大学）；The Chinese University of Hong Kong（香港中文大学）；Xi’an University of Electronic Science and Technology（西安电子科技大学）；University of California, Davis（加州大学戴维斯分校）；The University of Cambridge（剑桥大学）；LOGS AI
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence（北京交通大数据挖掘与具身智能重点实验室）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；State Key Laboratory of Advanced Rail Autonomous Operation（先进轨道交通自主运行国家重点实验室）
