arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.01136cs.CLcs.AIcs.LG
收藏

预训练神经语言模型提高急诊室分类模型的样本效率

Pre-training A Neural Language Model Improves The Sample Efficiency of an Emergency Room Classification Model

University of Bordeaux(波尔多大学);Bordeaux Population Health Research Center(波尔多人口健康研究中心);INSERM(法国国家健康与医学研究院);University Hospital of Bordeaux(波尔多大学医院);Inria BSO(法国国家信息与自动化研究所波尔多-西南研究中心)

Binbin Xu, Cédric Gil-Jardiné, Frantz Thiessard, Eric Tellier, Marta Avalos, Emmanuel Lagarde

摘要:

为了建立一个法国国家电子损伤监测系统的基础上急诊室访问,我们的目标是开发一个编码系统,以分类其原因,从临床笔记中的自由文本。监督学习技术在这一领域已经显示出良好的效果,但需要大量的专家注释的数据集,这是耗时和昂贵的获得。我们假设自然语言处理Transformer模型结合了生成式自监督预训练步骤,可以显著减少监督微调所需的注释样本数量。在这项初步研究中,我们测试我们的假设在预测访问是否是一个创伤性事件的后果或不从自由文本的临床笔记的简化问题。使用完全重新训练的GPT-2模型(没有OpenAI预训练的权重),我们评估了在监督学习任务之前应用具有未标记笔记的自监督预训练阶段的增益。结果表明,当应用预训练时,实现ginve性能水平(AUC>0.95)所需的数据数量减少了10倍。也就是说,对于16倍以上的数据,完全监督模型在AUC上实现了<1%的改善。总而言之,可以采用多用途神经语言模型(如GPT-2)来创建一个功能强大的工具,用于仅使用少量标记样本对自由文本笔记进行分类。

英文摘要:

To build a French national electronic injury surveillance system based on emergency room visits, we aim to develop a coding system to classify their causes from clinical notes in free-text. Supervised learning techniques have shown good results in this area but require a large amount of expert annotated dataset which is time consuming and costly to obtain. We hypothesize that the Natural Language Processing Transformer model incorporating a generative self-supervised pre-training step can significantly reduce the required number of annotated samples for supervised fine-tuning. In this preliminary study, we test our hypothesis in the simplified problem of predicting whether a visit is the consequence of a traumatic event or not from free-text clinical notes. Using fully re-trained GPT-2 models (without OpenAI pre-trained weights), we assess the gain of applying a self-supervised pre-training phase with unlabeled notes prior to the supervised learning task. Results show that the number of data required to achieve a ginve level of performance (AUC>0.95) was reduced by a factor of 10 when applying pre-training. Namely, for 16 times more data, the fully-supervised model achieved an improvement <1% in AUC. To conclude, it is possible to adapt a multi-purpose neural language model such as the GPT-2 to create a powerful tool for classification of free-text notes with only a small number of labeled samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑