Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models
Northwestern University（西北大学）；University of Science and Technology of China（中国科学技术大学）；National Yang Ming Chiao Tung University（阳明交通大学）；Simons Institute UC Berkeley（伯克利Simons研究所）
Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo, Zhao Song, Han Liu
备注：
Accepted at ICLR 2025. v2 matches the camera-ready version
We study the computational limits of Low-Rank Adaptation (LoRA) for finetuning transformer-based models using fine-grained complexity theory. Our key observation is that the existence of low-rank decompositions within the gradient computation of LoRA adaptation leads to possible algorithmic speedup. This allows us to (i) identify a phase transition behavior of efficiency assuming the Strong Exponential Time Hypothesis (SETH), and (ii) prove the existence of almost linear algorithms by controlling the LoRA update computation term by term. For the former, we identify a sharp transition in the efficiency of all possible rank-$r$ LoRA update algorithms for transformers, based on specific norms resulting from the multiplications of the input sequence $X$, pretrained weights ${W^\star}$, and adapter matrices $αB A/r$. Specifically, we derive a shared upper bound threshold for such norms, and show that efficient (sub-quadratic) approximation algorithms of LoRA exist only below this threshold. For the latter, we prove the existence of almost linear approximation algorithms for LoRA adaptation by utilizing the hierarchical low-rank structures of LoRA gradients and approximating the gradients with a series of chained low-rank approximations. To showcase our theory, we consider two practical scenarios: partial (e.g., only $W_V$ and $W_Q$) and full adaptations (e.g., $W_Q$, $W_V$, and $W_K$) of weights in attention heads.
Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
Georgia Institute of Technology（佐治亚理工学院）；National Taiwan University（国立台湾大学）；University of Palermo（帕尔米奥大学）；NVIDIA Research（NVIDIA研究）；Hon-Hai Quantum Computing Research Center（鸿海量子计算研究中心）
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, grea…
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China（软件研究所智能软件研究中心，中国科学院，北京）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京）
Texas A&M University（德克萨斯A&M大学）；Brookhaven National Laboratory（布鲁克海文国家实验室）
Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation (C-LoRA) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by develop…
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach
Tsinghua University（清华大学）；Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Institute for AI Industry Research (AIR)（人工智能产业研究院）；Shanghai AI Lab（上海人工智能实验室）
Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
School of Electronics and Computer Science, University of Southampton, UK（索姆塞特大学电子与计算机科学学院）；Department of Computer Science, University of Liverpool, UK（利弗pool大学计算机科学系）
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
School of Data, Mathematical and Statistical Sciences, University of Central Florida, United States（中佛罗里达大学数据、数学与统计科学学院）；College of Computing, Mohammed VI Polytechnic University (UM6P), Morocco（穆罕默德六世理工大学计算机学院）；Department of Computer Science, University of Central Florida, United States（中佛罗里达大学计算机科学系）
Rapid LoRA Aggregation for Wireless Channel Adaptation in Open-Set Radio Frequency Fingerprinting
School of Internet of Things, Nanjing University of Posts and Telecommunications（南京邮电大学物联网学院）；School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；National Mobile Communications Research Lab, Southeast University（东南大学国家移动通信研究中心）
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
City University of Hong Kong（香港城市大学）；Harvard University（哈佛大学）；Xi’an Jiaotong University（西安交通大学）；Tencent AI Lab（腾讯AI实验室）；Zhejiang University（浙江大学）；Pengcheng Laboratory（鹏城实验室）
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney（澳大利亚机器人中心，航空航天、机械与机电工程学院，悉尼大学）；Google DeepMind（谷歌DeepMind）
Privacy-Preserving Low-Rank Adaptation against Membership Inference Attacks for Latent Diffusion Models
The University of Auckland（奥克兰大学）；The National University of Singapore（新加坡国立大学）；The University of Melbourne（墨尔本大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology（华中科技大学）；Shenzhen Technology University（深圳技术大学）；City University of Hong Kong（香港城市大学）；Shenzhen University（深圳大学）
Federated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray Cohorts
North Carolina A&T State University（北卡罗来纳农工州立大学）；Tribhuvan University（特里布文大学）；University of Tennessee-Knoxville（田纳西大学诺克斯维尔分校）；University of South Dakota（南达科他大学）
Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints
Beijing Institute of Technology, Zhuhai（北京理工大学珠海学院）；The Hong Kong Polytechnic University（香港理工大学）；The University of Osaka（大阪大学）；Shenzhen University（深圳大学）
