GatorTron：一个大型临床语言模型，用于从非结构化电子健康记录中提取患者信息
GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records
摘要：
人们对开发人工智能（AI）系统来处理和解释电子健康记录（EHR）的兴趣越来越大。由预训练的语言模型驱动的自然语言处理（NLP）是利用临床叙述的医疗AI系统的关键技术。然而，临床语言模型很少，在临床领域中训练的最大的语言模型相对较小，只有1.1亿个参数（与一般领域中的数十亿个参数相比）。目前尚不清楚具有数十亿参数的大型临床语言模型如何帮助医疗AI系统利用非结构化EHR。在这项研究中，我们从头开始开发了一个大型临床语言模型- GatorTron -使用超过900亿字的文本（包括超过820亿字的去识别临床文本），并在5个临床NLP任务上对其进行了系统评估，包括临床概念提取，医学关系抽取，语义文本相似性，自然语言推理（NLI）和医学问答（MQA）。我们研究了（1）扩大参数的数量和（2）扩大训练数据的大小如何使这些NLP任务受益。GatorTron模型将临床语言模型的参数从1.1亿扩展到89亿，并改进了5项临床NLP任务（例如，NLI和MQA的准确性提高了9.6%和9.5%），可应用于医疗AI系统，以改善医疗保健服务。GatorTron模型可在https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og上公开获取。
英文摘要：
There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model - GatorTron - using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve 5 clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
