arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.20320cs.CVcs.AIcs.LG
收藏

MTLoRA:一种用于高效多任务学习的低秩适配方法

MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning

Brown University(布朗大学)

Ahmed Agiza, Marina Neseem, Sherief Reda

AI总结:

本文提出MTLoRA框架,通过任务无关与任务特定低秩适配模块解耦多任务学习微调参数空间,在PASCAL数据集上以更少可训练参数超越全量微调和现有参数高效方法。

摘要:

将在大规模数据集上预训练的模型适配到各种下游任务是深度学习中的常见策略。因此,参数高效微调方法已成为一种有前景的方式,可以在仅训练极少数量参数的情况下将预训练模型适配到不同任务。虽然这些方法大多针对单任务适配而设计,但多任务学习(MTL)架构中的参数高效训练仍未得到探索。在本文中,我们提出了MTLoRA,这是一个用于MTL模型参数高效训练的新颖框架。MTLoRA采用任务无关(Task-Agnostic)和任务特定(Task-Specific)低秩适配(Low-Rank Adaptation)模块,有效地解耦了MTL微调中的参数空间,从而使模型能够在MTL语境中熟练地同时处理任务专门化与任务交互。我们将MTLoRA应用于基于层次Transformer的MTL架构,将其适配到多个下游密集预测任务。我们在PASCAL数据集上的大量实验表明,与完全微调MTL模型相比,MTLoRA在下游任务上取得了更高的准确率,同时将可训练参数数量减少了3.6倍。此外,MTLoRA在可训练参数数量与下游任务准确率之间建立了帕累托最优(Pareto-optimal)权衡,在准确率和效率两方面均优于当前最先进的参数高效训练方法。我们的代码已公开。

英文摘要:

Adapting models pre-trained on large-scale datasets to a variety of downstream tasks is a common strategy in deep learning. Consequently, parameter-efficient fine-tuning methods have emerged as a promising way to adapt pre-trained models to different tasks while training only a minimal number of parameters. While most of these methods are designed for single-task adaptation, parameter-efficient training in Multi-Task Learning (MTL) architectures is still unexplored. In this paper, we introduce MTLoRA, a novel framework for parameter-efficient training of MTL models. MTLoRA employs Task-Agnostic and Task-Specific Low-Rank Adaptation modules, which effectively disentangle the parameter space in MTL fine-tuning, thereby enabling the model to adeptly handle both task specialization and interaction within MTL contexts. We applied MTLoRA to hierarchical-transformer-based MTL architectures, adapting them to multiple downstream dense prediction tasks. Our extensive experiments on the PASCAL dataset show that MTLoRA achieves higher accuracy on downstream tasks compared to fully fine-tuning the MTL model while reducing the number of trainable parameters by 3.6x. Furthermore, MTLoRA establishes a Pareto-optimal trade-off between the number of trainable parameters and the accuracy of the downstream tasks, outperforming current state-of-the-art parameter-efficient training methods in both accuracy and efficiency. Our code is publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑