In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices $\bA$ and $\bB$, but also propagate the learning to the pre-trained weights $\bW$ via updates utilizing the delta of the product of two low-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\bB^{(t)}$). Such a strategy effectively addresses the limitation that the incremental update of low-rank matrices is inadequate for learning representations capable for downstream tasks. Moreover, as the update of $\bW$ does not need to compute the gradients of $\bW$ and store their momentums, Delta-LoRA shares comparable memory requirements and computational costs with LoRA. Extensive experiments show that Delta-LoRA significantly outperforms existing low-rank adaptation methods. We further support these results with comprehensive analyses that underscore the effectiveness of Delta-LoRA.
Selective Aggregation for Low-Rank Adaptation in Federated Learning
The University of Hong Kong（香港大学）；Stanford University（斯坦福大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Materials Innovation Institute for Life Sciences and Energy (MILES), HKU-SIRI（香港大学-SIRI生命科学与能源材料创新研究所）
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（知识驱动人机智能工程研究中心，教育部，中国）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University（南京大学）
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Science, VNU-HCM（越南国家大学胡志明市分校科学大学）；Vietnam National University, Ho Chi Minh City（越南国家大学胡志明市分校）；Trivita AI
Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
Georgia Institute of Technology（佐治亚理工学院）；National Taiwan University（国立台湾大学）；University of Palermo（帕尔米奥大学）；NVIDIA Research（NVIDIA研究）；Hon-Hai Quantum Computing Research Center（鸿海量子计算研究中心）
School of Computing University of Connecticut（计算机学院 美国康涅狄格大学）；NEC Labs America（NEC美国实验室）；Dept. of Computer Science Georgia State University（计算机科学系 美国佐治亚州立大学）
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Harvard University（哈佛大学）；Xi’an University of Science and Technology（西安科技大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；University of Chicago（芝加哥大学）；Yunnan University（云南大学）；Tsinghua University（清华大学）
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney（澳大利亚机器人中心，航空航天、机械与机电工程学院，悉尼大学）；Google DeepMind（谷歌DeepMind）
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
Peking University（北京大学）；XiaoMi AI Lab（小米AI实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Electronic Science and Technology of China（电子科技大学）
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China（软件研究所智能软件研究中心，中国科学院，北京）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京）
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；National University of Singapore（新加坡国立大学）；Guangdong OPPO Mobile Telecommunications Corp., Ltd.（广东欧珀移动通信有限公司）
FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Shanghai Jiao Tong University（上海交通大学）；Queen's University Belfast（女王大学贝尔法斯特分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Purdue University（普渡大学）；Rice University（里奇大学）
