arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2005.00247cs.CL
收藏

AdapterFusion:面向迁移学习的非破坏性任务合成

AdapterFusion: Non-Destructive Task Composition for Transfer Learning

Technical University of Darmstadt(达姆施塔特工业大学);New York University(纽约大学)

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych

摘要:

顺序微调和多任务学习是旨在整合来自多个任务的知识的方法;然而,它们遭受灾难性遗忘和数据集平衡的困难。为了解决这些缺点,我们提出了AdapterFusion,一个新的两阶段学习算法,利用多个任务的知识。首先,在知识提取阶段,我们学习任务特定的参数,称为适配器,封装任务特定的信息。然后,我们在一个单独的知识组成步骤中组合适配器。我们表明,通过分离两个阶段,即,知识提取和知识组合,分类器可以有效地利用从多个任务中学习到的表示,以非破坏性的方式。我们对16个不同的NLU任务进行了经验评估,发现它有效地结合了模型不同层的各种类型的知识。我们表明,我们的方法优于传统的策略,如全微调以及多任务学习。我们的代码和适配器可以在AdapterHub.ml上找到。

英文摘要:

Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑