MIT（麻省理工学院）；Xyla（Xyla公司）；IBM Research（IBM研究院）；The Hospital for Sick Children（病童医院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J. Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, Emily Alsentzer
Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.
Samsung Research（三星研究院）；Seoul National University（首尔大学）
While deep learning techniques have shown promising results in many natural language processing (NLP) tasks, it has not been widely applied to the clinical domain. The lack of large datasets and the pervasive use of domain-specific language (i.e. abbreviations and acronyms) in the clinical domain causes slower progress in NLP tasks than that of the general NLP tasks. To fill this gap, we employ word/subword-level based models that adopt large-scale data-driven methods such as pre-trained language models and transfer learning in analyzing text for the clinical domain. Empirical results demonstr…
East China University of Science and Technology（华东理工大学）；Ruijin Hospital, School of Medicine, Shanghai Jiao Tong University（上海交通大学医学院附属瑞金医院）
Clinical text structuring is a critical and fundamental task for clinical research. Traditional methods such as taskspecific end-to-end models and pipeline models usually suffer from the lack of dataset and error propagation. In this paper, we present a question answering based clinical text structuring (QA-CTS) task to unify different specific tasks and make dataset shareable. A novel model that aims to introduce domain-specific features (e.g., clinical named entity information) into pre-trained language model is also proposed for QA-CTS task. Experimental results on Chinese pathology reports…
Clinical Natural Language Processing (NLP) has become an emerging technology in healthcare that leverages a large amount of free-text data in electronic health records (EHRs) to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models usually requires large annotated datasets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated dataset…
Northwestern University（西北大学）；Feinberg School of Medicine（范伯格医学院）
Acute kidney injury (AKI) is a common clinical syndrome characterized by a sudden episode of kidney failure or kidney damage within a few hours or a few days. Accurate early prediction of AKI for patients in ICU who are more likely than others to have AKI can enable timely interventions, and reduce the complications of AKI. Much of the clinical information relevant to AKI is captured in clinical notes that are largely unstructured text and requires advanced natural language processing (NLP) for useful information extraction. On the other hand, pre-trained contextual language models such as Bid…
Text Mining Unit Barcelona Supercomputing Center（文本挖掘组 巴塞罗那超级计算中心）
This work presents biomedical and clinical language models for Spanish by experimenting with different pretraining choices, such as masking at word and subword level, varying the vocabulary size and testing with domain data, looking for better language representations. Interestingly, in the absence of enough clinical data to train a model from scratch, we applied mixed-domain pretraining and cross-domain transfer approaches to generate a performant bio-clinical model suitable for real-world clinical data. We evaluated our models on Named Entity Recognition (NER) tasks for biomedical documents …
Using Large Language Models to Generate Clinical Trial Tables and Figures
Novartis Pharmaceuticals Corporation（诺华制药公司）；School of Biomedical Informatics The University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心生物医学信息学院）；Novartis Pharma AG（诺华制药公司）；The University of Texas MD Anderson Cancer Center（德克萨斯大学 MD 安德森癌症中心）
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University School of Medicine（同济大学医学院）；Sun Yat-sen University（中山大学）；Fuzhou University（福州大学）；Shenzhen Loop Area Institute（深圳河套学院）
Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
NYU Langone Health（纽约大学朗格尼医学中心）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；New York University（纽约大学）
Large language models are vulnerable to incidental information in clinical documentation and reasoning
NYU Langone Health（纽约大学朗格尼健康中心）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；University of Alabama at Birmingham Heersink School of Medicine（阿拉巴马大学伯明翰分校希尔斯金医学院）；Perelman School of Medicine at the University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；New York University（纽约大学）
Objective: Clinical knowledge enriched transformer models (e.g., ClinicalBERT) have state-of-the-art results on clinical NLP (natural language processing) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (e.g., Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long…
NIHR Oxford Health Biomedical Research Centre（NIHR牛津健康生物医学研究中心）；University of Oxford（牛津大学）
Prompt learning is a new paradigm in the Natural Language Processing (NLP) field which has shown impressive performance on a number of natural language tasks with common benchmarking text datasets in full, few-shot, and zero-shot train-evaluation setups. Recently, it has even been observed that large but frozen pre-trained language models (PLMs) with prompt learning outperform smaller but fine-tuned models. However, as with many recent NLP trends, the performance of even the largest PLMs such as GPT-3 do not perform well on specialized domains (e.g. medical text), and the common practice to ac…
Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Software, Nanjing University of Information Science and Technology (NUIST)（南京信息工程大学软件学院）；Department of Orthopedic, Bahawal Victoria Hospital（巴哈瓦尔维多利亚医院骨科部门）
Assistance Publique – Hôpitaux de Paris（巴黎公共援助医院）；Sorbonne Université（索邦大学）；Inserm（法国国家健康与医学研究院）；Université Sorbonne Paris Nord（索邦巴黎北大学）；Centre de Recherche des Cordeliers（科德利埃研究中心）；INRIA（法国国家信息与自动化研究所）
Background Clinical studies using real-world data may benefit from exploiting clinical reports, a particularly rich albeit unstructured medium. To that end, natural language processing can extract relevant information. Methods based on transfer learning using pre-trained language models have achieved state-of-the-art results in most NLP applications; however, publicly available models lack exposure to speciality-languages, especially in the medical field. Objective We aimed to evaluate the impact of adapting a language model to French clinical reports on downstream medical NLP tasks. Methods W…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；University of Münster（明斯特大学）
A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate h…
The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Center for Neurodegenerative Diseases（香港神经退行性疾病中心）
Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains, introducing many powerful LMs such as bio-lm and BioELECTRA. However, the applicability of these methods to real clinical use cases is hindered, due to the limitation of pre-trained LMs in processing long textual data with thousands of words, which is a common length for a clinical note. In this work, we explore long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudin…
Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds
Zhejiang University（浙江大学）；Tencent Youtu Lab（腾讯优图实验室）；Jarvis Research Center（贾维斯研究中心）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Deep learning algorithms are dependent on the availability of large-scale annotated clinical text datasets. The lack of such publicly available datasets is the biggest bottleneck for the development of clinical Natural Language Processing(NLP) systems. Zero-Shot Learning(ZSL) refers to the use of deep learning models to classify instances from new classes of which no training data have been seen before. Prompt-based learning is an emerging ZSL technique where we define task-based templates for NLP tasks. We developed a novel prompt-based clinical NLP framework called HealthPrompt and applied t…
Performance and Practical Considerations of Large and Small Language Models in Clinical Decision Support in Rheumatology
FOM University of Applied Sciences（FOM应用科技大学）；University Hospital Düsseldorf（杜塞尔多夫大学医院）；Heinrich-Heine-University（海因里希·海涅大学）；Hiller Research Center（希勒研究中心）；KEM Kliniken Essen-Mitte（埃森中部KEM医院）
Large Language Models as Unified Multimodal Learners for Clinical Prediction
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Charité Universitätsmedizin Berlin（柏林夏里特大学医学中心）；DNC Information Management GmbH（DNC信息管理有限公司）；Klinik für Akut- und Notfallmedizin, Asklepios Klinikum Harburg（哈堡阿斯克勒庇俄斯医院急性与急诊医学科）；Technical University Berlin（柏林工业大学）
