Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.
3DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selection
School of Computer Science, Peking University, Beijing, China(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室);National Engineering Research Center for Software Engineering, Peking University, China(软件工程国家级工程研究中心);Big Data Technology Research Center, Nanhu Laboratory, Jiaxing, China(南湖实验室大数据技术研究中心);Peking University Information Technology Institute, Tianjin Binhai, China(北京大学信息科学技术研究院)
SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASR
Department of Electrical Engineering KU Leuven Leuven, Belgium(电气工程系 卡尔·冯·特里尔大学 布鲁塞尔,比利时);Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA(语言技术研究所 哈佛大学 贝尔蒙特,美国)
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of International Business and Economics, China(对外经济贸易大学);XiaoMi AI Lab, China(小米AI实验室)
SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models
Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University(华南师范大学阿伯丁数据科学与人工智能学院);Department of Computing Science, University of Aberdeen(阿伯丁大学计算科学系)
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
Dalian University of Technology(大连理工大学);Northeastern University(东北大学);Hebei Key Laboratory of Marine Perception Network and Data Processing(河北省海洋感知网络与数据处理重点实验室);Nanjing Tech University(南京工业大学);Institute of Intelligent Manufacturing(智能制造研究院)