arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2208.12367cs.CLcs.LG
收藏

一种紧凑的神经语言模型预训练方法

A Compact Pretraining Approach for Neural Language Models

University of Arizona(亚利桑那大学);Harvard Medical School(哈佛医学院)

Shahriar Golchin, Mihai Surdeanu, Nazgol Tavabi, Ata Kiapour

摘要:

大型神经语言模型(NLM)的领域自适应在预训练阶段与大量非结构化数据相结合。然而,在这项研究中,我们表明,预训练的NLM从集中在域中的关键信息的数据的紧凑子集中更有效,更快地学习域中的信息。我们使用抽象摘要和提取关键字的组合从非结构化数据构建这些紧凑的子集。特别是,我们依靠BART来生成抽象摘要,KeyBERT从这些摘要中提取关键字(或直接从原始的非结构化文本中提取)。我们使用六种不同的设置来评估我们的方法:三个数据集与两个不同的NLM相结合。我们的研究结果表明,在使用我们的方法预训练的NLM上训练的特定任务分类器优于基于传统预训练的方法,即,对整个数据进行随机掩蔽,以及没有预训练的方法。此外,我们表明,与普通预训练相比,我们的策略将预训练时间减少了五倍。我们所有实验的代码都可以在https://github.com/shahriargolchin/compact-pretraining上公开获得。

英文摘要:

Domain adaptation for large neural language models (NLMs) is coupled with massive amounts of unstructured data in the pretraining phase. In this study, however, we show that pretrained NLMs learn in-domain information more effectively and faster from a compact subset of the data that focuses on the key information in the domain. We construct these compact subsets from the unstructured data using a combination of abstractive summaries and extractive keywords. In particular, we rely on BART to generate abstractive summaries, and KeyBERT to extract keywords from these summaries (or the original unstructured text directly). We evaluate our approach using six different settings: three datasets combined with two distinct NLMs. Our results reveal that the task-specific classifiers trained on top of NLMs pretrained using our method outperform methods based on traditional pretraining, i.e., random masking on the entire data, as well as methods without pretraining. Further, we show that our strategy reduces pretraining time by up to five times compared to vanilla pretraining. The code for all of our experiments is publicly available at https://github.com/shahriargolchin/compact-pretraining.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑