State of the art models using deep neural networks have become very good in learning an accurate mapping from inputs to outputs. However, they still lack generalization capabilities in conditions that differ from the ones encountered during training. This is even more challenging in specialized, and knowledge intensive domains, where training data is limited. To address this gap, we introduce MedNLI - a dataset annotated by doctors, performing a natural language inference task (NLI), grounded in the medical history of patients. We present strategies to: 1) leverage transfer learning using datasets from the open domain, (e.g. SNLI) and 2) incorporate domain knowledge from external data and lexical sources (e.g. medical terminologies). Our results demonstrate performance gains using both strategies.
Synapse Développement；IRIT, University of Toulouse（图卢兹大学IRIT研究所）；CNRS（法国国家科学研究中心）
Various NLP problems -- such as the prediction of sentence similarity, entailment, and discourse relations -- are all instances of the same general task: the modeling of semantic relations between a pair of textual elements. A popular model for such problems is to embed sentences into fixed size vectors, and use composition functions (e.g. concatenation or sum) of those vectors as features for the prediction. At the same time, composition of embeddings has been a main focus within the field of Statistical Relational Learning (SRL) whose goal is to predict relations between entities (typically …
Crowdworker-constructed natural language inference (NLI) datasets have been found to contain statistical artifacts associated with the annotation process that allow hypothesis-only classifiers to achieve better-than-random performance (Poliak et al., 2018; Gururanganet et al., 2018; Tsuchiya, 2018). We investigate whether MedNLI, a physician-annotated dataset with premises extracted from clinical notes, contains such artifacts (Romanov and Shivade, 2018). We find that entailed hypotheses contain generic versions of specific concepts in the premise, as well as modifiers related to responsivenes…
Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校巴卡尔计算健康科学研究所）；Center for Data-driven Insights and Innovation, University of California, Office of the President（加州大学校长办公室数据驱动洞察与创新中心）；University of California, San Francisco（加州大学旧金山分校）
Several biomedical language models have already been developed for clinical language inference. However, these models typically utilize general vocabularies and are trained on relatively small clinical corpora. We sought to evaluate the impact of using a domain-specific vocabulary and a large clinical training corpus on the performance of these language models in clinical language inference. We trained a Bidirectional Encoder Decoder from Transformers (BERT) model using a diverse, deidentified corpus of 75 million deidentified clinical notes authored at the University of California, San Franci…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Deep Neural Network (DNN) models have been shown to have high empirical privacy leakages. Clinical language models (CLMs) trained on clinical data have been used to improve performance in biomedical natural language processing tasks. In this work, we investigate the risks of training-data leakage through white-box or black-box access to CLMs. We design and employ membership inference attacks to estimate the empirical privacy leaks for model architectures like BERT and GPT2. We show that membership inference attacks on CLMs lead to non-trivial privacy leakages of up to 7%. Our results show that…
We submitted two systems to the SemEval-2016 Task 12: Clinical TempEval challenge, participating in Phase 1, where we identified text spans of time and event expressions in clinical notes and Phase 2, where we predicted a relation between an event and its parent document creation time. For temporal entity extraction, we find that a joint inference-based approach using structured prediction outperforms a vanilla recurrent neural network that incorporates word embeddings trained on a variety of large clinical document sets. For document creation time relations, we find that a combination of date…
Samsung Research（三星研究院）；Seoul National University（首尔大学）
While deep learning techniques have shown promising results in many natural language processing (NLP) tasks, it has not been widely applied to the clinical domain. The lack of large datasets and the pervasive use of domain-specific language (i.e. abbreviations and acronyms) in the clinical domain causes slower progress in NLP tasks than that of the general NLP tasks. To fill this gap, we employ word/subword-level based models that adopt large-scale data-driven methods such as pre-trained language models and transfer learning in analyzing text for the clinical domain. Empirical results demonstr…
Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Boston Children’s Hospital Harvard Medical School（波士顿儿童医院哈佛医学院）；University of Colorado Anschutz（科罗拉多大学安舒茨分校）；Loyola University Chicago（芝加哥洛约拉大学）
School of Engineering and Applied Science Aston University（阿斯顿大学工程与应用科学学院）
Small and Medium Enterprises (SMEs) now generate digital data at an unprecedented rate from online transactions, social media marketing and associated customer interactions, online product or service reviews and feedback, clinical diagnosis, Internet of Things (IoT) sensors, and production processes. All these forms of data can be transformed into monetary value if put into a proper data value chain. This requires both skills and IT investments for the long-term benefit of businesses. However, such spending is beyond the capacity of most SMEs due to their limited resources and restricted acces…
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about the information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role rever…
University of California, Los Angeles（加州大学洛杉矶分校）；University of Southern California（南加州大学）
There has been a steady need in the medical community to precisely extract the temporal relations between clinical events. In particular, temporal information can facilitate a variety of downstream applications such as case report retrieval and medical question answering. Existing methods either require expensive feature engineering or are incapable of modeling the global relational dependencies among the events. In this paper, we propose a novel method, Clinical Temporal ReLation Exaction with Probabilistic Soft Logic Regularization and Global Inference (CTRL-PG) to tackle the problem at the …
AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；University of Delaware, Newark, DE, USA（特拉华大学）
Pontifical Catholic University of Paraná（巴拉那天主教大学）；Philips Research North America（飞利浦北美研究院）
The high volume of research focusing on extracting patient's information from electronic health records (EHR) has led to an increase in the demand for annotated corpora, which are a very valuable resource for both the development and evaluation of natural language processing (NLP) algorithms. The absence of a multi-purpose clinical corpus outside the scope of the English language, especially in Brazilian Portuguese, is glaring and severely impacts scientific progress in the biomedical NLP field. In this study, we developed a semantically annotated corpus using clinical texts from multiple medi…
School of Electronic Engineering & Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；Centre for Trauma Sciences, Blizard Institute, Queen Mary University of London（伦敦玛丽女王大学布利扎德研究所创伤科学中心）；Royal London Hospital, Barts and The London NHS Trust（皇家伦敦医院）；Academic Department of Military Surgery and Trauma, Royal Centre for Defence Medicine（皇家国防医学中心军事外科与创伤学术部）
Various AI models are increasingly being considered as part of clinical decision-support tools. However, the trustworthiness of such models is rarely considered. Clinicians are more likely to use a model if they can understand and trust its predictions. Key to this is if its underlying reasoning can be explained. A Bayesian network (BN) model has the advantage that it is not a black-box and its reasoning can be explained. In this paper, we propose an incremental explanation of inference that can be applied to hybrid BNs, i.e. those that contain both discrete and continuous nodes. The key quest…
How do we most effectively treat a disease or condition? Ideally, we could consult a database of evidence gleaned from clinical trials to answer such questions. Unfortunately, no such database exists; clinical trial results are instead disseminated primarily via lengthy natural language articles. Perusing all such articles would be prohibitively time-consuming for healthcare practitioners; they instead tend to depend on manually compiled systematic reviews of medical literature to inform care. NLP may speed this process up, and eventually facilitate immediate consult of published evidence. The…
Bridging AI Innovation and Healthcare Needs: Lessons Learned from Incorporating Modern NLP at The BC Cancer Registry
British Columbia Cancer Registry, Provincial Health Services Authority, Vancouver, Canada（不列颠哥伦比亚省癌症登记处、省卫生服务局、温哥华，加拿大）；The Data Science Institute, University of British Columbia, Vancouver, Canada（数据科学研究所、不列颠哥伦比亚大学、温哥华，加拿大）；School of Population and Public Health, University of British Columbia, Vancouver, Canada（流行病学与公共卫生学院、不列颠哥伦比亚大学、温哥华，加拿大）
本文基于在BC Cancer Registry实施NLP模型的经验，指出医疗AI部署需以业务目标为导向、采用迭代开发与跨学科协作，并关注数据质量与错误缓解，为医疗组织应用AI/NLP提供实践指导。
CCS: Clinical Consensus Selection for Radiology Report Generation
School of Computing Science, University of Glasgow（格拉斯哥大学计算机科学学院）；School of Electrical and Computer Engineering, University of Sydney（悉尼大学电气与计算机工程学院）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
De-identification is not enough: a comparison between de-identified and synthetic clinical notes
Department of Computer Science, University of Manitoba（曼尼托巴大学计算机科学系）；McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心麦克威廉姆斯生物医学信息学院）
Enhancing Trustworthy Clinical Diagnosis Decision-Making in Large Language Models via Etiology-Aware Attention Supervision
Zhejiang University（浙江大学）；University of Chinese Academy of Sciences（中国科学院大学）；First Affiliated Hospital, College of Medicine, Zhejiang University（浙江大学医学院附属第一医院）
Learning to represent free text is a core task in many clinical machine learning (ML) applications, as clinical text contains observations and plans not otherwise available for inference. State-of-the-art methods use large language models developed with immense computational resources and training data; however, applying these models is challenging because of the highly varying syntax and vocabulary in clinical free text. Structured information such as International Classification of Disease (ICD) codes often succinctly abstracts the most important facts of a clinical encounter and yields good…
Feinberg School of Medicine（范伯格医学院）；Northwestern University（西北大学）
Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce…
Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration expert domain knowledge. In this work, we introduced UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was perfor…
