Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis
Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, USA(工程科学与应用数学系,西北大学,埃文斯顿,美国);Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, USA(工业工程与管理科学系,西北大学,埃文斯顿,美国)
Previous empirical studies have shown that LoRA achieves accuracy comparable to full-parameter methods on downstream fine-tuning tasks, even for rank-1 updates. By contrast, the theoretical underpinnings of the dependence of LoRA's accuracy on update rank remain relatively unexplored. In this work, we compare the accuracy of rank-r LoRA updates against full-parameter updates for fine-tuning tasks from a dynamical systems perspective. We perform gradient flow analysis in both full-rank and low-rank regimes to establish explicit relationships between rank and accuracy for two loss functions under LoRA. While gradient flow equations for LoRA are presented in prior work, we rigorously derive their form and show that they are identical for simultaneous and sequential LoRA parameter updates. We then use the resulting dynamical system equations to obtain closed-form relationships between LoRA rank and accuracy for trace-squared and Frobenius-norm low-rank approximation loss functions.
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学,上海,中国);Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China(柔性医疗机器人上海市重点实验室,同仁医院,医疗机器人研究所,上海交通大学,上海,中国)
Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints
Beijing Institute of Technology, Zhuhai(北京理工大学珠海学院);The Hong Kong Polytechnic University(香港理工大学);The University of Osaka(大阪大学);Shenzhen University(深圳大学)
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
City University of Hong Kong(香港城市大学);Harvard University(哈佛大学);Xi’an Jiaotong University(西安交通大学);Tencent AI Lab(腾讯AI实验室);Zhejiang University(浙江大学);Pengcheng Laboratory(鹏城实验室)
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Harvard University(哈佛大学);Xi’an University of Science and Technology(西安科技大学);University of Chinese Academy of Sciences(中国科学院大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);University of Chicago(芝加哥大学);Yunnan University(云南大学);Tsinghua University(清华大学)
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
Selective Aggregation for Low-Rank Adaptation in Federated Learning
The University of Hong Kong(香港大学);Stanford University(斯坦福大学);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Materials Innovation Institute for Life Sciences and Energy (MILES), HKU-SIRI(香港大学-SIRI生命科学与能源材料创新研究所)
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学与技术学院);Artificial Intelligence Research Institute, iFLYTEK Company Ltd.(iFLYTEK公司人工智能研究院)
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology(华中科技大学);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学);Shenzhen University(深圳大学)
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China(软件研究所智能软件研究中心,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)