arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.15082cs.CLcs.LG
收藏

PEMT:多任务相关性引导的专家混合实现参数高效迁移学习

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning

Zhejiang University(浙江大学);Salesforce Research Asia(Salesforce 亚洲研究院);State Street Technology (Zhejiang) Ltd.(道富科技(浙江)有限公司)

Zhisheng Lin, Han Fu, Chenghao Liu, Zhuo Li, Jianling Sun

AI总结:

PEMT提出基于多任务迁移学习的参数高效微调框架,通过专家混合和任务稀疏性损失捕获源任务与目标任务的相关性,在17个数据集上超越全量微调及现有方法。

摘要:

参数高效微调(PEFT)已成为将预训练语言模型高效适配到各种任务的有效方法。近年来,人们越来越关注从一个或多个任务向目标下游任务迁移知识,以实现性能提升。然而,当前方法通常要么在单个任务上训练适配器,要么从源任务中蒸馏共享知识,未能充分利用任务特定知识以及源任务与目标任务之间的相关性。为克服这些局限,我们提出了PEMT,一种基于多任务迁移学习的新型参数高效微调框架。PEMT扩展了专家混合(MoE)框架,将可迁移知识捕获为在源任务上训练的适配器的加权组合。这些权重由一个门控单元确定,该单元利用任务描述提示向量衡量目标任务与每个源任务之间的相关性。为充分利用任务特定知识,我们还提出了任务稀疏性损失(Task Sparsity Loss),以提高门控单元的稀疏性。我们在17个数据集上的广泛任务中进行了实验。实验结果表明,我们的PEMT相比全量微调以及最先进的PEFT和知识迁移方法,在各种任务上均取得了稳定的提升。结果凸显了我们方法的有效性,它能够充分挖掘跨多个任务的知识和相关性特征。

英文摘要:

Parameter-efficient fine-tuning (PEFT) has emerged as an effective method for adapting pre-trained language models to various tasks efficiently. Recently, there has been a growing interest in transferring knowledge from one or multiple tasks to the downstream target task to achieve performance improvements. However, current approaches typically either train adapters on individual tasks or distill shared knowledge from source tasks, failing to fully exploit task-specific knowledge and the correlation between source and target tasks. To overcome these limitations, we propose PEMT, a novel parameter-efficient fine-tuning framework based on multi-task transfer learning. PEMT extends the mixture-of-experts (MoE) framework to capture the transferable knowledge as a weighted combination of adapters trained on source tasks. These weights are determined by a gated unit, measuring the correlation between the target and each source task using task description prompt vectors. To fully exploit the task-specific knowledge, we also propose the Task Sparsity Loss to improve the sparsity of the gated unit. We conduct experiments on a broad range of tasks over 17 datasets. The experimental results demonstrate our PEMT yields stable improvements over full fine-tuning, and state-of-the-art PEFT and knowledge transferring methods on various tasks. The results highlight the effectiveness of our method which is capable of sufficiently exploiting the knowledge and correlation features across multiple tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑