Parameter-efficient fine-tuning (PEFT) is a flexible and efficient method for adapting large language models (LLMs) to downstream tasks. Among these methods, weight-decomposed low-rank adaptation (DoRA) is a promising approach that decomposes weight matrices into magnitude and direction components to mimic full fine-tuning (FT) better. However, DoRA's simultaneous optimization of these components makes it over-expressive, increases the risk of overfitting, and creates a coupled updating pattern that limits its learning capacity. To address these issues, we propose Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), a novel PEFT method based on a bi-level optimization framework. BiDoRA fundamentally differs from DoRA by optimizing the magnitude and direction in two separate, asynchronous loops using distinct training and validation data splits. This decoupled optimization process effectively mitigates overfitting and allows for more flexible updates that align even more closely with FT. For instance, weight decomposition analysis shows BiDoRA achieves a magnitude-direction update correlation of $-8.042$, significantly closer to the FT ideal compared to $-1.784$ for DoRA. Evaluation of BiDoRA on diverse tasks spanning natural language understanding, generation, token classification, and extremely small biomedical datasets reveals that it consistently outperforms DoRA and a wide range of leading PEFT methods. This improvement is statistically significant, as demonstrated on the GLUE benchmark where BiDoRA surpasses DoRA with a p-value of $2.4\times10^{-4}$ in terms of the Wilcoxon signed-rank test. The code for BiDoRA is available at https://github.com/t2ance/BiDoRA.
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of International Business and Economics, China(对外经济贸易大学);XiaoMi AI Lab, China(小米AI实验室)
3DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selection
School of Computer Science, Peking University, Beijing, China(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室);National Engineering Research Center for Software Engineering, Peking University, China(软件工程国家级工程研究中心);Big Data Technology Research Center, Nanhu Laboratory, Jiaxing, China(南湖实验室大数据技术研究中心);Peking University Information Technology Institute, Tianjin Binhai, China(北京大学信息科学技术研究院)
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(胡志明市国家大学科学大学);Vietnam National University, Ho Chi Minh City(胡志明市越南国家大学);Trivita AI
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China(北京大学人工智能研究院);Yuanpei College, Peking University, Beijing, China(北京大学元培学院);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(通用人工智能国家重点实验室);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳))
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));University of Sydney(悉尼大学)
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
Domain Adaptation of Automated Treatment Planning from Computed Tomography to Magnetic Resonance
University of Toronto(多伦多大学);Techna Institute, University Health Network(大学健康网络泰克纳研究所);Princess Margaret Cancer Center(玛格丽特公主癌症中心);Peter Munk Cardiac Center, University Health Network(大学健康网络彼得·蒙克心脏中心);Joint Department of Medical Imaging, University Health Network(大学健康网络医学影像联合部);Vector Institute(矢量研究所)
Efficient Parameter Adaptation for Multi-Modal Medical Image Segmentation and Prognosis
Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence(计算机视觉系,Mohamed bin Zayed人工智能大学);Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence(机器学习系,Mohamed bin Zayed人工智能大学);Michigan State University(密歇根州立大学);M42 Health(M42健康)