arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00573cs.CL
收藏

CoMoL: 通过动态核心空间融合实现高效的LoRA专家混合

CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

Zhejiang University(浙江大学);Wechat, Tencent(微信,腾讯)

Jie Cao, Zhenxuan Fan, Zhuonan Wang, Tianwei Lin, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, Feifei Shao, Hongwei Wang, Jun Xiao, Siliang Tang

AI总结:

本文提出CoMoL,一种新的MoE-LoRA框架,通过引入核心空间专家和核心空间路由,实现参数高效和细粒度适应,同时在多个任务中优于现有方法。

摘要:

大型语言模型(LLMs)通过参数高效微调(PEFT)在多样化的下游和领域特定任务中取得显著性能。然而,现有的PEFT方法,特别是MoE-LoRA架构,由于LoRA专家和实例级路由的普及,存在参数效率低和粗粒度适应的问题。为了解决这些问题,我们提出了核心空间混合的LoRA(CoMoL),一种新颖的MoE-LoRA框架,结合了专家多样性、参数效率和细粒度适应。具体而言,CoMoL引入了两个关键组件:核心空间专家和核心空间路由。核心空间专家将每个专家存储在紧凑的核心矩阵中,保留多样性同时控制参数增长。核心空间路由动态选择并激活每个标记的适当核心专家,实现细粒度、输入自适应的路由。激活的核心专家通过软融合策略合并成一个核心专家,再与共享的LoRA结合形成专用的LoRA模块。此外,路由网络被投影到与LoRA矩阵相同的低秩空间中,进一步减少参数开销而不影响表达能力。广泛的实验表明,CoMoL保留了MoE-LoRA架构的适应性,同时在参数效率上与标准LoRA相当,在多个任务中持续优于现有方法。

英文摘要:

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameter-efficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instance-level routing. To address these issues, we propose Core Space Mixture of LoRA (\textbf{CoMoL}), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: core space experts and core space routing. Core space experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core space routing dynamically selects and activates the appropriate core experts for each token, enabling fine-grained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL retains the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to standard LoRA, consistently outperforming existing methods across multiple tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑