BIOptimus：利用课程学习预训练最优生物医学语言模型以用于命名实体识别
BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition
AI总结：
本文系统比较生物医学语言模型预训练策略，提出上下文化权重蒸馏并结合课程学习训练 BIOptimus，在多项生物医学 NER 任务上取得新最优结果。
摘要：
在大型语料库上以自监督方式预训练语言模型（LM），然后针对下游任务进行微调，已有助于解决命名实体识别（NER）等监督学习任务中标签数据有限的问题。近期生物医学语言处理研究提供了许多采用不同方法和技术预训练的生物医学 LM，这些模型推进了包括 NER 在内的许多 BioNLP 任务的结果。然而，目前仍缺乏对在生物医学领域中能够更优运行的预训练方法的全面比较。本文旨在研究不同的预训练方法，例如从头开始预训练生物医学 LM，以及以持续方式进行预训练。我们将现有方法与本文提出的预训练方法进行比较：该方法在发现新 token 的上下文中，通过蒸馏来自 BERT 模型的现有权重来初始化新 token 的权重。该方法有助于加快预训练阶段并提升 NER 性能。此外，我们还比较了掩码率、破坏策略和掩码策略如何影响生物医学 LM 的性能。最后，利用实验得到的洞见，我们提出了一种新的生物医学 LM（BIOptimus），它采用课程学习（CL）和上下文化权重蒸馏方法进行预训练。我们的模型在若干生物医学命名实体识别（NER）任务上创造了新的最优水平。我们发布了代码和所有预训练模型。
英文摘要：
Using language models (LMs) pre-trained in a self-supervised setting on large corpora and then fine-tuning for a downstream task has helped to deal with the problem of limited label data for supervised learning tasks such as Named Entity Recognition (NER). Recent research in biomedical language processing has offered a number of biomedical LMs pre-trained using different methods and techniques that advance results on many BioNLP tasks, including NER. However, there is still a lack of a comprehensive comparison of pre-training approaches that would work more optimally in the biomedical domain. This paper aims to investigate different pre-training methods, such as pre-training the biomedical LM from scratch and pre-training it in a continued fashion. We compare existing methods with our proposed pre-training method of initializing weights for new tokens by distilling existing weights from the BERT model inside the context where the tokens were found. The method helps to speed up the pre-training stage and improve performance on NER. In addition, we compare how masking rate, corruption strategy, and masking strategies impact the performance of the biomedical LM. Finally, using the insights from our experiments, we introduce a new biomedical LM (BIOptimus), which is pre-trained using Curriculum Learning (CL) and contextualized weight distillation method. Our model sets new states of the art on several biomedical Named Entity Recognition (NER) tasks. We release our code and all pre-trained models
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
