面向多任务学习的带基于LoRA的专家混合的自适应共享专家
Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
AI总结：
针对现有MoE-MTL方法单任务转多任务时适配冗余、知识共享低效的问题，提出带自适应共享专家的LoRA基MoE及细粒度专家设计，在PASCAL-Context基准上验证了性能提升效果。
摘要：
专家混合（Mixture-of-Experts, MoE）已成为多任务学习（multi-task learning, MTL）的强大框架。然而，现有MoE-MTL方法通常依赖单任务预训练骨干网络，在从单任务学习向多任务学习（STL到MTL）过渡期间存在适配冗余、知识共享效率低的问题。为解决这些局限，我们在基于低秩适配（low-rank adaptation, LoRA）的MoE中提出自适应共享专家（adaptive shared experts, ASE），其中共享专家被分配由路由计算的、与稀疏专家联合归一化的门控权重。该设计促进了STL到MTL的过渡，增强了专家的专业性与协作性。此外，我们通过增加LoRA专家数量同时按比例降低其秩引入细粒度专家，在相当的参数预算下实现更有效的知识共享。在PASCAL-Context基准上、统一训练设置下开展的大量实验表明，ASE在多种配置下持续提升性能，验证了细粒度设计对MTL的有效性。
英文摘要：
Mixture-of-Experts (MoE) has emerged as a powerful framework for multi-task learning (MTL). However, existing MoE-MTL methods often rely on single-task pretrained backbones and suffer from redundant adaptation and inefficient knowledge sharing during the transition from single-task to multi-task learning (STL to MTL). To address these limitations, we propose adaptive shared experts (ASE) within a low-rank adaptation (LoRA) based MoE, where shared experts are assigned router-computed gating weights jointly normalized with sparse experts. This design facilitates STL to MTL transition, enhances expert specialization, and cooperation. Furthermore, we incorporate fine-grained experts by increasing the number of LoRA experts while proportionally reducing their rank, enabling more effective knowledge sharing under a comparable parameter budget. Extensive experiments on the PASCAL-Context benchmark, under unified training settings, demonstrate that ASE consistently improves performance across diverse configurations and validates the effectiveness of fine-grained designs for MTL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
