arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.12943cs.CLcs.LG
收藏

一种简单高效的任务自适应文本分类预训练方法

Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification

Pune Institute of Computer Technology(浦那计算机技术学院);L3Cube;Indian Institute of Technology Madras(印度理工学院马德拉斯分校)

Arnav Ladkat, Aamir Miyajiwala, Samiksha Jagadale, Rekha Kulkarni, Raviraj Joshi

摘要:

语言模型使用大型通用数据语料库(如图书语料库、常见爬行和维基百科)进行预训练,这对于模型理解语言的语言特征至关重要。新的研究建议使用领域自适应预训练(DAPT)和任务自适应预训练(TAPT)作为最终微调任务之前的中间步骤。这一步有助于覆盖目标领域词汇表,并提高下游任务的模型性能。在这项工作中,我们研究了仅训练嵌入层对TAPT和特定任务微调期间模型性能的影响。基于我们的研究,我们提出了一种简单的方法,使基于BERT的模型的TAPT的中间步骤更有效地通过执行BERT层的选择性预训练。我们表明,在TAPT期间仅训练BERT嵌入层足以适应目标域的词汇并实现相当的性能。我们的方法在计算上是有效的,在TAPT期间训练的参数减少了78%。所提出的嵌入层微调方法也可以是有效的领域自适应技术。

英文摘要:

Language models are pre-trained using large corpora of generic data like book corpus, common crawl and Wikipedia, which is essential for the model to understand the linguistic characteristics of the language. New studies suggest using Domain Adaptive Pre-training (DAPT) and Task-Adaptive Pre-training (TAPT) as an intermediate step before the final finetuning task. This step helps cover the target domain vocabulary and improves the model performance on the downstream task. In this work, we study the impact of training only the embedding layer on the model's performance during TAPT and task-specific finetuning. Based on our study, we propose a simple approach to make the intermediate step of TAPT for BERT-based models more efficient by performing selective pre-training of BERT layers. We show that training only the BERT embedding layer during TAPT is sufficient to adapt to the vocabulary of the target domain and achieve comparable performance. Our approach is computationally efficient, with 78\% fewer parameters trained during TAPT. The proposed embedding layer finetuning approach can also be an efficient domain adaptation technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑