arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.08124cs.CL
收藏

何时使用多任务学习与中间微调进行预训练编码器迁移学习

When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning

Johns Hopkins University(约翰斯·霍普金斯大学);Brigham Young University(杨百翰大学);Microsoft Semantic Machines(微软语义机器公司)

Orion Weller, Kevin Seppi, Matt Gardner

摘要:

近年来,自然语言处理(NLP)中的迁移学习(TL)引起了人们的极大兴趣,因为预先训练的模型已经显示出令人印象深刻的迁移到新任务的能力。在微调期间使用多个监督数据集已经出现了三种主要策略:在目标任务上训练之前训练中间任务(STILTs),使用多任务学习(MTL)联合训练补充任务和目标任务(成对MTL),或者简单地使用MTL联合训练所有可用的数据集(MTL-ALL)。在这项工作中,我们比较了所有三种TL方法在一个全面的分析GLUE数据集套件。我们发现,有一个简单的启发式当使用这些技术之一,在其他:成对MTL是优于STILTs时,目标任务的实例比支持任务,反之亦然。我们表明,这在GLUE数据集上92%以上的适用情况下都是正确的,并通过不同数据集大小的实验验证了这一假设。这种启发式方法的简单性和有效性令人惊讶,值得TL社区进行更多的探索。此外,我们发现MTL-ALL在几乎所有情况下都比成对方法更差。我们希望这项研究能够帮助其他人在自然语言处理任务中选择TL方法。

英文摘要:

Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL) to train jointly on a supplementary task and the target task (pairwise MTL), or simply using MTL to train jointly on all available datasets (MTL-ALL). In this work, we compare all three TL methods in a comprehensive analysis on the GLUE dataset suite. We find that there is a simple heuristic for when to use one of these techniques over the other: pairwise MTL is better than STILTs when the target task has fewer instances than the supporting task and vice versa. We show that this holds true in more than 92% of applicable cases on the GLUE dataset and validate this hypothesis with experiments varying dataset size. The simplicity and effectiveness of this heuristic is surprising and warrants additional exploration by the TL community. Furthermore, we find that MTL-ALL is worse than the pairwise methods in almost every case. We hope this study will aid others as they choose between TL methods for NLP tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑