DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
City University of Hong Kong(香港城市大学);Tsinghua University(清华大学);Peking University(北京大学);University of Michigan(密歇根大学);University of Chinese Academy of Sciences(中国科学院大学)
Guanzhi Deng, Bo Li, Ronghao Chen, Xiujin Liu, Zhuo Han, Huacan Wang, Lijie Wen, Linqi Song
Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning methods, such as LoRA, are widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches typically assign identical LoRA ranks to all expert modules, ignoring the heterogeneous specialization of pretrained experts. This uniform allocation leads to a resource mismatch: task-relevant experts are under-provisioned, while less relevant ones receive redundant parameters. To address this, we propose DR-LoRA, a Dynamic Rank LoRA framework for fine-tuning pretrained MoE models. Specifically, DR-LoRA initializes all expert LoRA modules with a small active rank and uses an expert saliency score, which combines routing frequency and gradient-based rank importance, to identify which experts would benefit most from additional capacity. It then periodically expands the active ranks of the task-critical expert LoRA, progressively constructing a heterogeneous rank distribution tailored to the target task. Experiments on three MoE models across six tasks show that DR-LoRA consistently outperforms LoRA and other strong baselines, demonstrating that task-adaptive heterogeneous rank allocation is an effective strategy to improve active capacity utilization in MoE fine-tuning.
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
School of Computer Science and Technology, Beijing Jiaotong University, China(北京交通大学计算机科学与技术学院);Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China(广西trusted软件重点实验室,桂林电子科技大学);Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China(北京交通数据挖掘与具身智能重点实验室);Institute of AI for Industries, Chinese Academy of Sciences, China(工业人工智能研究院,中国科学院);Nanjing Institute of Software Technology, China(南京软件技术研究院)
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Zhejiang University(浙江大学);Shanghai Innovation Institute(上海创新研究院);ByteDance Inc.(字节跳动公司);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院(BAAI));School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong(香港大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of California, Los Angeles(加州大学洛杉矶分校);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(中国科学院计算技术研究所);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Experts
School of Information and Network Security, People’s Public Security University of China(中国人民公安大学信息与网络安全学院);Key Laboratory of Security Prevention Technology and Risk Assessment of the Ministry of Public Security(公安部安全防范技术与风险评估重点实验室);School of Criminal Investigation, People’s Public Security University of China(中国人民公安大学刑事侦查学院)
Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
Yanshan University(燕山大学);Delft University of Technology(代尔夫特理工大学);Shanghai University of Finance and Economics(上海财经大学);A*STAR Centre for Frontier AI Research(新加坡科技研究局前沿人工智能研究中心);Nanyang Technological University(南洋理工大学)
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Auditory Cognition and Computational Acoustics Lab(听觉认知与计算声学实验室);MoE Key Lab of Artificial Intelligence(人工智能MoE重点实验室);AI Institute School of Computer Science, Shanghai Jiao Tong University(计算机学院人工智能研究所,上海交通大学);Tencent AI Lab(腾讯AI实验室)
Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
School of Information and Software Engineering, University of Electronic Science and Technology of China(信息与软件工程学院,电子科学与技术大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学)
P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(东华大学人工智能教育研究所);Xiaohongshu Inc.(小红书公司)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
Customizing Language Models with Instance-wise LoRA for Sequential Recommendation
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Electronic Science Research Institute of China Electronics Technology Group Corporation(中国电子科技集团公司电子科学研究院)
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学)