arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.07202cs.CL
收藏

医学 BERT 本地化与生物医学 BERT 增强的预训练技术

Pre-training technique to localize medical BERT and enhance biomedical BERT

Osaka University Graduate School of Medicine（大阪大学医学系研究科）；Osaka University（大阪大学）

Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Konishi, Jun Kamohara, Yasushi Matsumura

备注：
We made the pre-trained weights of ouBioBERT and the source code for fine-tuning freely available at https://github.com/sy-wada/blue_benchmark_with_transformers
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通过对领域语料上采样并扩充词汇表进行均衡同步预训练，以解决医学等小语料领域 BERT 训练困难问题，并在日语医学分类和 BLUE 基准上验证了模型性能提升。

摘要：

在原始文本上预训练大规模神经语言模型，对提升自然语言处理（NLP）中的迁移学习做出了重要贡献。随着基于 Transformer 的语言模型（如来自 Transformer 的双向编码器表示 BERT）的引入，NLP 从自由文本中进行信息抽取的性能在通用领域和医学领域都得到了显著提升；然而，对于公开可用且高质量、大规模数据库较少的领域，很难训练出表现良好的专用 BERT 模型。我们假设可以通过对领域特定语料库进行上采样，并以均衡方式将其与更大规模语料库一起用于预训练来解决这一问题。我们提出的方法由单一干预和一个选项组成：上采样和扩充词汇表之后的同步预训练。我们进行了三项实验并评估了所得模型。我们确认，在医学文档分类任务上，我们的日语医学 BERT 优于传统基线和其它 BERT 模型；并且，我们同时使用通用领域和医学领域语料库预训练的英语 BERT，在生物医学语言理解评估（BLUE）基准上表现足够好，可用于实际应用。此外，我们增强后的生物医学 BERT 模型在预训练期间未使用临床记录，其在 BLUE 基准上的临床分数和生物医学分数均比未使用我们提出方法训练的消融模型高出 0.3 分。通过上采样来自适合目标任务语料库的实例进行均衡预训练，使我们能够构建高性能的 BERT 模型。

英文摘要：

Pre-training large-scale neural language models on raw texts has made a significant contribution to improving transfer learning in natural language processing (NLP). With the introduction of transformer-based language models, such as bidirectional encoder representations from transformers (BERT), the performance of information extraction from a free text by NLP has significantly improved for both the general domain and medical domain; however, it is difficult to train specific BERT models that perform well for domains in which there are few publicly available databases of high quality and large size. We hypothesized that this problem can be addressed by up-sampling a domain-specific corpus and using it for pre-training with a larger corpus in a balanced manner. Our proposed method consists of a single intervention with one option: simultaneous pre-training after up-sampling and amplified vocabulary. We conducted three experiments and evaluated the resulting products. We confirmed that our Japanese medical BERT outperformed conventional baselines and the other BERT models in terms of the medical document classification task and that our English BERT pre-trained using both the general and medical-domain corpora performed sufficiently well for practical use in terms of the biomedical language understanding evaluation (BLUE) benchmark. Moreover, our enhanced biomedical BERT model, in which clinical notes were not used during pre-training, showed that both the clinical and biomedical scores of the BLUE benchmark were 0.3 points above that of the ablation model trained without our proposed method. Well-balanced pre-training by up-sampling instances derived from a corpus appropriate for the target task allows us to construct a high-performance BERT model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑