arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.11332cs.CL
收藏

AdaptSum:面向低资源领域自适应的抽象摘要

AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

Center for Artificial Intelligence Research (CAiRE)(人工智能研究中心(CAiRE));Department of Electronic and Computer Engineering(电子及计算机工程学系);The Hong Kong University of Science and Technology(香港科技大学)

Tiezheng Yu, Zihan Liu, Pascale Fung

摘要:

现有的抽象摘要模型通常依赖于大量的标记数据,这降低了它们在这些数据不可用的领域的泛化能力。在本文中,我们提出了一个研究的领域自适应的抽象摘要任务在六个不同的目标领域在低资源设置。具体来说,我们在三种不同的设置下研究了大规模生成模型的第二阶段预训练:1)源域预训练; 2)域自适应预训练; 3)任务自适应预训练。实验表明,预训练的有效性与预训练数据与目标域任务的相似度有关。此外,我们发现持续的预训练可能会导致预训练模型的灾难性遗忘,而遗忘较少的学习方法可以缓解这个问题。此外,结果表明,低资源和高资源设置之间仍然存在巨大的差距,这突出了需要更先进的领域自适应方法的抽象摘要任务。

英文摘要:

State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑