LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
Parameter-efficient fine-tuning (PEFT), particularly Low-Rank Adaptation (LoRA), adapts large language models (LLMs) by training only a small fraction of parameters. However, as the rank of the low-rank matrices used for adaptation increases, LoRA often exhibits an unstable "double descent" phenomenon, characterized by transient divergence in the training loss, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. To address this, we introduce LoRA-MGPO, a framework that incorporates Momentum-Guided Perturbation Optimization (MGPO). MGPO stabilizes training dynamics by mitigating the double descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer's state, thus avoiding dual gradient computations. Additionally, an adaptive normalization scheme scales the magnitude of perturbations based on an exponential moving average (EMA) of gradient norms, further enhancing stability. While EMA controls the magnitude of the perturbations, MGPO guides their direction, ensuring a more stable optimization trajectory. Experiments on a suite of natural language understanding and generation benchmarks show that LoRA-MGPO consistently achieves superior performance over LoRA and other PEFT methods. The analysis indicates that LoRA-MGPO leads to smoother loss curves, faster convergence, and improved generalization by stabilizing the training process and mitigating the attraction to sharp minima.
Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
Yanshan University(燕山大学);Delft University of Technology(代尔夫特理工大学);Shanghai University of Finance and Economics(上海财经大学);A*STAR Centre for Frontier AI Research(新加坡科技研究局前沿人工智能研究中心);Nanyang Technological University(南洋理工大学)
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Shanghai AI Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);University of Sydney(悉尼大学)
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.(浪潮 Genersoft 有限公司);Inspur Group Co. Ltd.(浪潮集团有限公司);Interactive Entertainment Group, Tencent Inc(腾讯公司互动娱乐集团);Nanjing University of Information Science and Technology(南京信息工程大学);Northeastern University(东北大学)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation
Applied Artificial Intelligence and Intelligent Systems (AAIINS) Laboratory(应用人工智能与智能系统实验室);Department of Computer Science and Engineering(计算机科学与工程系);Department of Data Science and Artificial Intelligence(数据科学与人工智能系);Department of Software Engineering(软件工程系);Faculty of Science and Information Technology(科学与信息技术学院);Faculty of Science and Technology(科学与技术学院)
Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University(吉林大学符号计算与知识工程重点实验室);RIKEN Center for Advanced Intelligence Project(RIKEN高级智能项目中心);Graduate School of Frontier Sciences, University of Tokyo(东京大学前沿科学研究生院)