arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04823cs.AIcs.CL
收藏

DR-LoRA:动态秩LoRA用于混合专家模型的微调

DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models

City University of Hong Kong(香港城市大学);Tsinghua University(清华大学);Peking University(北京大学);University of Michigan(密歇根大学);University of Chinese Academy of Sciences(中国科学院大学)

Guanzhi Deng, Bo Li, Ronghao Chen, Xiujin Liu, Zhuo Han, Huacan Wang, Lijie Wen, Linqi Song

AI总结:

DR-LoRA通过动态分配不同秩的LoRA模块,提升混合专家模型微调效果,实验表明其优于传统方法。

摘要:

混合专家(MoE)已成为扩展大型语言模型(LLMs)的重要范式。参数高效微调方法,如LoRA,被广泛用于适应预训练的MoE LLMs以适应下游任务。然而,现有方法通常将相同的LoRA秩分配给所有专家模块,忽略了预训练专家的异质专业化。这种统一分配导致资源不匹配:任务相关的专家被过度分配,而相关性较低的专家则被冗余参数所占用。为此,我们提出DR-LoRA,一种动态秩LoRA框架,用于微调预训练的MoE模型。具体而言,DR-LoRA初始化所有专家LoRA模块为小的活跃秩,并利用专家显著性分数,结合路由频率和基于梯度的秩重要性,以确定哪些专家最能从额外容量中受益。然后,它会定期扩展任务关键专家LoRA的活跃秩,逐步构建适合目标任务的异质秩分布。在六个任务上对三种MoE模型的实验表明,DR-LoRA在多个方面优于LoRA和其他强基线,证明了任务自适应的异质秩分配是提高MoE微调中活跃容量利用率的有效策略。

英文摘要:

Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning methods, such as LoRA, are widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches typically assign identical LoRA ranks to all expert modules, ignoring the heterogeneous specialization of pretrained experts. This uniform allocation leads to a resource mismatch: task-relevant experts are under-provisioned, while less relevant ones receive redundant parameters. To address this, we propose DR-LoRA, a Dynamic Rank LoRA framework for fine-tuning pretrained MoE models. Specifically, DR-LoRA initializes all expert LoRA modules with a small active rank and uses an expert saliency score, which combines routing frequency and gradient-based rank importance, to identify which experts would benefit most from additional capacity. It then periodically expands the active ranks of the task-critical expert LoRA, progressively constructing a heterogeneous rank distribution tailored to the target task. Experiments on three MoE models across six tasks show that DR-LoRA consistently outperforms LoRA and other strong baselines, demonstrating that task-adaptive heterogeneous rank allocation is an effective strategy to improve active capacity utilization in MoE fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑