arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.03241cs.CLcs.AIcs.LGcs.NE
收藏

语言模型的持续预训练

Continual Pre-training of Language Models

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);KDDI Research(KDDI综合研究所);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)

Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, Bing Liu

AI总结:

本文针对语言模型持续领域自适应预训练中的灾难性遗忘问题,提出基于软掩码机制、通用知识保留代理和表示对比知识整合的新方法,在多个无标注领域语料上实现知识迁移并提升下游任务性能。

摘要:

语言模型(LMs)在自然语言处理的快速发展中发挥了重要作用。本文研究语言模型的持续预训练,特别是持续领域自适应预训练(或称持续 DAP 训练)。已有研究表明,使用领域语料库进一步预训练语言模型以使其适应该领域,可以提升该领域内的下游任务性能。本文提出了一种新方法,利用一系列无标注领域语料库对语言模型进行持续 DAP 训练,使其适应这些领域,从而提升相应下游任务的性能。我们方法的核心创新是一种软掩码机制,它直接控制对语言模型的更新。同时,本文还提出了一种新的代理指标,用于保留原始语言模型中的通用知识。此外,该方法通过对比先前学习到的领域知识表示(包括预训练语言模型中的通用知识)与当前完整网络中的知识表示,实现知识整合。该方法不仅克服了灾难性遗忘,还实现了知识迁移,从而提升下游任务性能。实证评估证明了所提方法的有效性。

英文摘要:

Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑