arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.00784cs.CL
收藏

高效多领域语言模型预训练的实证研究

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training

Amazon(亚马逊)

Kristjan Arumae, Qing Sun, Parminder Bhatia

摘要:

预训练大语言模型已经成为自然语言处理社区的标准。这些模型是在通用数据(例如BookCorpus和英文维基百科)上预先训练的,并且经常在同一领域的任务上进行微调。然而,为了在诸如临床命名实体识别和关系抽取之类的领域外任务上实现最先进的性能,需要附加的领域内预训练。在实践中,分阶段的多域预训练在基于诸如GLUE的通用基准进行评估时以灾难性遗忘(CF)的形式呈现性能恶化。在本文中,我们进行了实证调查,以减轻CF的已知方法。我们发现,弹性重量巩固提供了最好的整体成绩,在7个通用任务的性能只下降了0.33%,同时在生物医学任务中保持竞争力。此外,我们探索基于梯度和潜在聚类的数据选择技术,以提高覆盖率时,使用弹性权重合并和经验重放方法。

英文摘要:

Pre-training large language models has become a standard in the natural language processing community. Such models are pre-trained on generic data (e.g. BookCorpus and English Wikipedia) and often fine-tuned on tasks in the same domain. However, in order to achieve state-of-the-art performance on out of domain tasks such as clinical named entity recognition and relation extraction, additional in domain pre-training is required. In practice, staged multi-domain pre-training presents performance deterioration in the form of catastrophic forgetting (CF) when evaluated on a generic benchmark such as GLUE. In this paper we conduct an empirical investigation into known methods to mitigate CF. We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. Furthermore, we explore gradient and latent clustering based data selection techniques to improve coverage when using elastic weight consolidation and experience replay methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑