Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models
New York University(纽约大学);University of Minnesota(明尼苏达大学);Tulane University(杜兰大学);The Chinese University of Hong Kong(香港中文大学);Stevens Institute of Technology(史蒂文斯理工学院)
This paper presents a novel methodology of fine-tuning for large language models-dynamic LoRA. Building from the standard Low-Rank Adaptation framework, this methodology further adds dynamic adaptation mechanisms to improve efficiency and performance. The key contribution of dynamic LoRA lies within its adaptive weight allocation mechanism coupled with an input feature-based adaptive strategy. These enhancements allow for a more precise fine-tuning process that is more tailored to specific tasks. Traditional LoRA methods use static adapter settings, not considering the different importance of model layers. In contrast, dynamic LoRA introduces a mechanism that dynamically evaluates the layer's importance during fine-tuning. This evaluation enables the reallocation of adapter parameters to fit the unique demands of each individual task, which leads to better optimization results. Another gain in flexibility arises from the consideration of the input feature distribution, which helps the model generalize better when faced with complicated and diverse datasets. The joint approach boosts not only the performance over each single task but also the generalization ability of the model. The efficiency of the dynamic LoRA was validated in experiments on benchmark datasets, such as GLUE, with surprising results. More specifically, this method achieved 88.1% accuracy with an F1-score of 87.3%. Noticeably, these improvements were made at a slight increase in computational costs: only 0.1% more resources than standard LoRA. This balance between performance and efficiency positions dynamic LoRA as a practical, scalable solution for fine-tuning LLMs, especially in resource-constrained scenarios. To take it a step further, its adaptability makes it a promising foundation for much more advanced applications, including multimodal tasks.
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology(华中科技大学);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学);Shenzhen University(深圳大学)
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
City University of Hong Kong(香港城市大学);Harvard University(哈佛大学);Xi’an Jiaotong University(西安交通大学);Tencent AI Lab(腾讯AI实验室);Zhejiang University(浙江大学);Pengcheng Laboratory(鹏城实验室)
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints
Beijing Institute of Technology, Zhuhai(北京理工大学珠海学院);The Hong Kong Polytechnic University(香港理工大学);The University of Osaka(大阪大学);Shenzhen University(深圳大学)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
ChunkWise LoRA: Adaptive Sequence Partitioning for Memory-Efficient Low-Rank Adaptation and Accelerated LLM Inference
Bentley University USA(伯克利大学);Cornell University USA(康奈尔大学);University of California Berkeley USA(加州大学伯克利分校);George Mason University USA(乔治·马歇尔大学);Georgia Institute of Technology USA(佐治亚理工学院)
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)