Learning structures of the French clinical language:development and validation of word embedding models using 21 million clinical reports from electronic health records
Assistance Publique – Hôpitaux de Paris（巴黎公共援助医院）；Sorbonne Université（索邦大学）；Inserm（法国国家健康与医学研究院）；Université Sorbonne Paris Nord（索邦巴黎北大学）；Centre de Recherche des Cordeliers（科德利埃研究中心）；INRIA（法国国家信息与自动化研究所）
Basile Dura, Charline Jean, Xavier Tannier, Alice Calliger, Romain Bey, Antoine Neuraz, Rémi Flicoteaux
Background
Clinical studies using real-world data may benefit from exploiting clinical reports, a particularly rich albeit unstructured medium. To that end, natural language processing can extract relevant information. Methods based on transfer learning using pre-trained language models have achieved state-of-the-art results in most NLP applications; however, publicly available models lack exposure to speciality-languages, especially in the medical field.
Objective
We aimed to evaluate the impact of adapting a language model to French clinical reports on downstream medical NLP tasks.
Methods
We leveraged a corpus of 21M clinical reports collected from August 2017 to July 2021 at the Greater Paris University Hospitals (APHP) to produce two CamemBERT architectures on speciality language: one retrained from scratch and the other using CamemBERT as its initialisation. We used two French annotated medical datasets to compare our language models to the original CamemBERT network, evaluating the statistical significance of improvement with the Wilcoxon test.
Results
Our models pretrained on clinical reports increased the average F1-score on APMed (an APHP-specific task) by 3 percentage points to 91%, a statistically significant improvement. They also achieved performance comparable to the original CamemBERT on QUAERO. These results hold true for the fine-tuned and from-scratch versions alike, starting from very few pre-training samples.
Conclusions
We confirm previous literature showing that adapting generalist pre-train language models such as CamenBERT on speciality corpora improves their performance for downstream clinical NLP tasks. Our results suggest that retraining from scratch does not induce a statistically significant performance gain compared to fine-tuning.
Austin Health（奥斯汀健康）；University of Melbourne（墨尔本大学）
We propose an approach for adapting the DeBERTa model for electronic health record (EHR) tasks using domain adaptation. We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (MeDeBERTa) and an XGBoost model. We evaluate performance on three benchmark tasks for emergency department outcomes using the MIMIC-IV-ED dataset. We preprocess the data to convert it into text format and generate fo…
Institut Supérieur d’Electronique de Paris（巴黎高等电子学院）；Université de Reims Champagne Ardenne（兰斯大学）
In the era of digitization, different actors in agriculture produce numerous data. Such data contains already latent historical knowledge in the domain. This knowledge enables us to precisely study natural hazards within global or local aspects, and then improve the risk prevention tasks and augment the yield, which helps to tackle the challenge of growing population and changing alimentary habits. In particular, French Plants Health Bulletins (BSV, for its name in French Bulletin de Sant{é} du V{é}g{é}tal) give information about the development stages of phytosanitary risks in agricultural pr…
PARHAF, a human-authored corpus of clinical reports for fictitious patients in French
Sorbonne Université, Université Sorbonne Paris Nord, Inserm, Limics（索邦大学，索邦大学巴黎北分校，法国国家医学研究院，Limics）；Université Paris-Saclay, UVSQ, Assistance Publique-Hôpitaux de Paris, Raymond Poincaré University Hospital, Infectious Disease Department（巴黎-萨克雷大学，UVSQ，巴黎公共医院，雷诺·帕洛西大学医院，传染病部门）；Yonsei University College of Medicine, Gangnam Severance Hospital, Department of Laboratory Medicine（延世大学医学院，江南松趾医院，实验室医学部）；Assistance Publique-Hôpitaux de Paris, Department of medical information（巴黎公共医院，医学信息部门）；Health Data Hub, 75015, Paris, France（健康数据枢纽，75015，法国巴黎）；Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France（巴黎-萨克雷大学，法国国家科学研究中心，LISN，91400，法国奥赛）；Université Paris-Dauphine, PSL, CNRS, CEREMADE, 75016, Paris, France（巴黎-第七大学，PSL，法国国家科学研究中心，CEREMADE，75016，法国巴黎）
Tokyo Metropolitan University（东京都立大学）；RIKEN（理化学研究所）；National Hospital Organization（国立医院机构）；Kitami Institute of Technology（北见工业大学）
Automated summarization of clinical texts can reduce the burden of medical professionals. "Discharge summaries" are one promising application of the summarization, because they can be generated from daily inpatient records. Our preliminary experiment suggests that 20-31% of the descriptions in discharge summaries overlap with the content of the inpatient records. However, it remains unclear how the summaries should be generated from the unstructured source. To decompose the physician's summarization process, this study aimed to identify the optimal granularity in summarization. We first define…
Spoken medical dialogue systems are increasingly attracting interest to enhance access to healthcare services and improve quality and traceability of patient care. In this paper, we focus on medical drug prescriptions acquired on smartphones through spoken dialogue. Such systems would facilitate the traceability of care and would free clinicians' time. However, there is a lack of speech corpora to develop such systems since most of the related corpora are in text form and in English. To facilitate the research and development of spoken medical dialogue systems, we present, to the best of our k…
ReLyfe - Medical Intelligence（ReLyfe 医疗智能公司）；Ecole Polytechnique（巴黎综合理工学院）；BeCareLink（BeCareLink 公司）
Structuring medical data in France remains a challenge mainly because of the lack of medical data due to privacy concerns and the lack of methods and approaches on processing the French language. One of these challenges is structuring drug-related information in French clinical documents. To our knowledge, over the last decade, there are less than five relevant papers that study French prescriptions. This paper proposes a new approach for extracting drug-related information from French clinical scanned documents while preserving patients' privacy. In addition, we deployed our method in a healt…
A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages
BIFOLD；TU Berlin（柏林工业大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Université Paris-Saclay（巴黎萨克雷大学）；Nara Institute of Science and Technology（奈良科学技术研究所）；RIKEN（理化学研究所）
Identifying Mentions of Pain in Mental Health Records Text: A Natural Language Processing Approach
Institute of Psychiatry, Psychology and Neurosciences, King’s College London（伦敦国王学院精神病学、心理学与神经科学研究所）；Health Data Research UK（英国健康数据研究中心）；South London and Maudsley Biomedical Research Centre（南伦敦和莫兹利生物医学研究中心）；The South London and Maudsley NHS Foundation Trust（南伦敦和莫兹利国民保健信托基金会）
Electronic Health Records (EHRs) have become the primary form of medical data-keeping across the United States. Federal law restricts the sharing of any EHR data that contains protected health information (PHI). De-identification, the process of identifying and removing all PHI, is crucial for making EHR data publicly available for scientific research. This project explores several deep learning-based named entity recognition (NER) methods to determine which method(s) perform better on the de-identification task. We trained and tested our models on the i2b2 training dataset, and qualitatively …
Dementia is under-recognized in the community, under-diagnosed by healthcare professionals, and under-coded in claims data. Information on cognitive dysfunction, however, is often found in unstructured clinician notes within medical records but manual review by experts is time consuming and often prone to errors. Automated mining of these notes presents a potential opportunity to label patients with cognitive concerns who could benefit from an evaluation or be referred to specialist care. In order to identify patients with cognitive concerns in electronic medical records, we applied natural la…
Electronic health records (EHRs), digital collections of patient healthcare events and observations, are ubiquitous in medicine and critical to healthcare delivery, operations, and research. Despite this central role, EHRs are notoriously difficult to process automatically. Well over half of the information stored within EHRs is in the form of unstructured text (e.g. provider notes, operation reports) and remains largely untapped for secondary use. Recently, however, newer neural network and deep learning approaches to Natural Language Processing (NLP) have made considerable advances, outperfo…
In the electronic health record, using clinical notes to identify entities such as disorders and their temporality (e.g. the order of an event relative to a time index) can inform many important analyses. However, creating training data for clinical entity tasks is time consuming and sharing labeled data is challenging due to privacy concerns. The information needs of the COVID-19 pandemic highlight the need for agile methods of training machine learning models for clinical notes. We present Trove, a framework for weakly supervised entity classification using medical ontologies and expert-gene…
Nanjing Medical University（南京医科大学）；the First Affiliated Hospital（第一附属医院）
Event extraction is an important work of medical text processing. According to the complex characteristics of medical text annotation, we use the end-to-end event extraction model to enhance the output formatting information of events. Through pre training and fine-tuning, we can extract the attributes of the four dimensions of medical text: anatomical position, subject word, description word and occurrence state. On the test set, the accuracy rate was 0.4511, the recall rate was 0.3928, and the F1 value was 0.42. The method of this model is simple, and it has won the second place in the task …
Validating transformers for redaction of text from electronic health records in real-world healthcare
King’s College London（伦敦国王学院）；King’s College Hospital（国王学院医院）；Guy’s and St Thomas’ NHS Foundation Trust（盖伊和圣托马斯国民保健服务基金会信托）；University College London（伦敦大学学院）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Accessing longitudinal multimodal Electronic Healthcare Records (EHRs) is challenging due to privacy concerns, which hinders the use of ML for healthcare applications. Synthetic EHRs generation bypasses the need to share sensitive real patient records. However, existing methods generate single-modal EHRs by unconditional generation or by longitudinal inference, which falls short of low flexibility and makes unrealistic EHRs. In this work, we propose to formulate EHRs generation as a text-to-text translation task by language models (LMs), which suffices to highly flexible event imputation durin…
Medical entity extraction (EE) is a standard procedure used as a first stage in medical texts processing. Usually Medical EE is a two-step process: named entity recognition (NER) and named entity normalization (NEN). We propose a novel method of doing medical EE from electronic health records (EHR) as a single-step multi-label classification task by fine-tuning a transformer model pretrained on a large EHR dataset. Our model is trained end-to-end in an distantly supervised manner using targets automatically extracted from medical knowledge base. We show that our model learns to generalize for …
