With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic search-free models with DyLoRA at least 4 to 7 times (depending to the task) faster than LoRA without significantly compromising performance. Moreover, our models can perform consistently well on a much larger range of ranks compared to LoRA.
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology(华中科技大学);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学);Shenzhen University(深圳大学)
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心);Helmholtz Munich(海德堡-穆恩医疗中心);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);MDSI(慕尼黑数据科学研究所)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
University of Science and Technology of China(中国科学技术大学);Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences(中国科学院智能机械研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Anhui Disaster Warning & Agrometeorological Information Center(安徽省灾害预警与农业气象信息中心)
Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education(交通大数据与人工智能重点实验室(北京交通大学),教育部);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
ML Research, Morgan Stanley, New York, NY 10019, USA(ML研究,摩根大通,纽约,纽约州,10019,美国);Visa Research, Austin, TX 78759, USA(Visa研究,奥斯汀,德克萨斯州,78759,美国);Department of ECE, University of Minnesota, MN 55455, USA(电子工程系,明尼苏达大学,明尼苏达州,55455,美国)
Chien-Shiung Wu College, Southeast University(陈希ung Wu学院,东南大学);MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University(人工智能MOE实验室,人工智能研究院,上海交通大学计算机科学学院);School of Computer Science and Engineering, Southeast University(计算机科学与工程学院,东南大学);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用关键实验室(东南大学),教育部,中国)
Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(越南国家大学胡志明市分校科学大学);Vietnam National University, Ho Chi Minh City(越南国家大学胡志明市分校);Trivita AI