Since clinical letters contain sensitive information, clinical-related datasets can not be widely applied in model training, medical research, and teaching. This work aims to generate reliable, various, and de-identified synthetic clinical letters. To achieve this goal, we explored different pre-trained language models (PLMs) for masking and generating text. After that, we worked on Bio\_ClinicalBERT, a high-performing model, and experimented with different masking strategies. Both qualitative and quantitative methods were used for evaluation. Additionally, a downstream task, Named Entity Recognition (NER), was also implemented to assess the usability of these synthetic letters.
The results indicate that 1) encoder-only models outperform encoder-decoder models. 2) Among encoder-only models, those trained on general corpora perform comparably to those trained on clinical data when clinical information is preserved. 3) Additionally, preserving clinical entities and document structure better aligns with our objectives than simply fine-tuning the model. 4) Furthermore, different masking strategies can impact the quality of synthetic clinical letters. Masking stopwords has a positive impact, while masking nouns or verbs has a negative effect. 5) For evaluation, BERTScore should be the primary quantitative evaluation metric, with other metrics serving as supplementary references. 6) Contextual information does not significantly impact the models' understanding, so the synthetic clinical letters have the potential to replace the original ones in downstream tasks.
Clinical Natural Language Processing (NLP) has become an emerging technology in healthcare that leverages a large amount of free-text data in electronic health records (EHRs) to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models usually requires large annotated datasets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated dataset…
A Dataset and Resources for Identifying Patient Health Literacy Information from Clinical Notes
National Library of Medicine（国家医学图书馆）；Massachusetts Institute of Technology（麻省理工学院）；Yonsei University College of Medicine（延世大学医学院）；Beth Israel Deaconess Medical Center（贝斯以色列医疗中心）；NYC Health + Hospitals - Lincoln（纽约市卫生局-林肯医院）
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
University of Luebeck（吕贝克大学）；University of Tübingen（图宾根大学）；University Hospital Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医院）；University Hospital Würzburg（维尔茨堡大学医院）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Genie Enterprise Deutschland GmbH
Objective: The 2022 n2c2 NLP Challenge posed identification of social determinants of health (SDOH) in clinical narratives. We present three systems that we developed for the Challenge and discuss the distinctive task formulation used in each of the three systems. Materials and Methods: The first system identifies target pieces of information independently using machine learning classifiers. The second system uses a large language model (LLM) to extract complete structured outputs per document. The third system extracts candidate phrases using machine learning and identifies target relations w…
Deep learning algorithms are dependent on the availability of large-scale annotated clinical text datasets. The lack of such publicly available datasets is the biggest bottleneck for the development of clinical Natural Language Processing(NLP) systems. Zero-Shot Learning(ZSL) refers to the use of deep learning models to classify instances from new classes of which no training data have been seen before. Prompt-based learning is an emerging ZSL technique where we define task-based templates for NLP tasks. We developed a novel prompt-based clinical NLP framework called HealthPrompt and applied t…
Radiology reports contain a diverse and rich set of clinical abnormalities documented by radiologists during their interpretation of the images. Comprehensive semantic representations of radiological findings would enable a wide range of secondary use applications to support diagnosis, triage, outcomes prediction, and clinical research. In this paper, we present a new corpus of radiology reports annotated with clinical findings. Our annotation schema captures detailed representations of pathologic findings that are observable on imaging ("lesions") and other types of clinical problems ("medica…
DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models
Premium Research Institute for Human Metaverse Medicine (WPI-PRIMe)（人类元宇宙医学高级研究机构（WPI-PRIMe））；Osaka University（大阪大学）；Department of Cardiology, The First Affiliated Hospital of Dalian Medical University（大连医科大学第一附属医院心内科）；Institute of Computing Technology, Chinese Academy of Science（中国科学院计算技术研究所）；D3 Center, Osaka University（大阪大学D3中心）；Agency for Science, Technology and Research (A*STAR)（科技研究局（A*STAR））
Objective: The generalizability of clinical large language models is usually ignored during the model development process. This study evaluated the generalizability of BERT-based clinical NLP models across different clinical settings through a breast cancer phenotype extraction task. Materials and Methods: Two clinical corpora of breast cancer patients were collected from the electronic health records from the University of Minnesota and the Mayo Clinic, and annotated following the same guideline. We developed three types of NLP models (i.e., conditional random field, bi-directional long short…
MIT（麻省理工学院）；Xyla（Xyla公司）；IBM Research（IBM研究院）；The Hospital for Sick Children（病童医院）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we…
ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment
University of Washington（华盛顿大学）；The Hospital for Sick Children（圣马可医院）；University of Bologna（博洛尼亚大学）；The Chinese University of Hong Kong（香港中文大学）；University of Saskatchewan（萨斯喀彻温大学）
Assistance Publique – Hôpitaux de Paris（巴黎公共援助医院）；Sorbonne Université（索邦大学）；Inserm（法国国家健康与医学研究院）；Université Sorbonne Paris Nord（索邦巴黎北大学）；Centre de Recherche des Cordeliers（科德利埃研究中心）；INRIA（法国国家信息与自动化研究所）
Background Clinical studies using real-world data may benefit from exploiting clinical reports, a particularly rich albeit unstructured medium. To that end, natural language processing can extract relevant information. Methods based on transfer learning using pre-trained language models have achieved state-of-the-art results in most NLP applications; however, publicly available models lack exposure to speciality-languages, especially in the medical field. Objective We aimed to evaluate the impact of adapting a language model to French clinical reports on downstream medical NLP tasks. Methods W…
A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification
Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校巴卡尔计算健康科学研究所）；Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；University of California, Berkeley（加州大学伯克利分校）；Center for Data-driven Insights and Innovation, University of California, Office of the President（加州大学校长办公室数据驱动洞察与创新中心）；Department of Pediatrics, University of California, San Francisco（加州大学旧金山分校儿科系）
Retrieval-Reasoning Large Language Model-based Synthetic Clinical Trial Generation
Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎- Rocquencourt 国家信息与自动化研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）；University of Chicago（芝加哥大学）；Stanford University（斯坦福大学）；Purdue University（普渡大学）；University of Southern California（南加州大学）
Coronavirus disease 2019 (COVID-19) is a global pandemic. Although much has been learned about the novel coronavirus since its emergence, there are many open questions related to tracking its spread, describing symptomology, predicting the severity of infection, and forecasting healthcare utilization. Free-text clinical notes contain critical information for resolving these questions. Data-driven, automatic information extraction models are needed to use this text-encoded information in large-scale studies. This work presents a new clinical corpus, referred to as the COVID-19 Annotated Clinica…
Pontifical Catholic University of Paraná（巴拉那天主教大学）；Philips Research North America（飞利浦北美研究院）
The high volume of research focusing on extracting patient's information from electronic health records (EHR) has led to an increase in the demand for annotated corpora, which are a very valuable resource for both the development and evaluation of natural language processing (NLP) algorithms. The absence of a multi-purpose clinical corpus outside the scope of the English language, especially in Brazilian Portuguese, is glaring and severely impacts scientific progress in the biomedical NLP field. In this study, we developed a semantically annotated corpus using clinical texts from multiple medi…
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Ricoh Software Research Center (Beijing)（理光软件研究所（北京））；School of Software, Harbin University of Science and Technology（哈尔滨理工大学软件学院）；Medical Records Room, the Second Affiliated Hospital of Harbin Medical University（哈尔滨医科大学附属第二医院病案室）
Objective: To build a comprehensive corpus covering syntactic and semantic annotations of Chinese clinical texts with corresponding annotation guidelines and methods as well as to develop tools trained on the annotated corpus, which supplies baselines for research on Chinese texts in the clinical domain. Materials and methods: An iterative annotation method was proposed to train annotators and to develop annotation guidelines. Then, by using annotation quality assurance measures, a comprehensive corpus was built, containing annotations of part-of-speech (POS) tags, syntactic tags, entities, as…
On the Effects of Low-Quality Training Data on Information Extraction from Clinical Reports
Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche（意大利国家研究委员会信息科学与技术研究所）；Qatar Computing Research Institute, Qatar Foundation（卡塔尔基金会卡塔尔计算研究所）；Consiglio Nazionale delle Ricerche（意大利国家研究委员会）
Brown University（布朗大学）；The Warren Alpert Medical School, Brown University（布朗大学沃伦·阿尔珀特医学院）；School of Public Health, Brown University（布朗大学公共卫生学院）；Providence VA Medical Center（普罗维登斯退伍军人事务医疗中心）；Rhode Island Quality Institute（罗德岛质量研究所）
Anginal symptoms can connote increased cardiac risk and a need for change in cardiovascular management. This study evaluated the potential to extract these symptoms from physician notes using the Bidirectional Encoder from Transformers language model fine-tuned on a domain-specific corpus. The history of present illness section of 459 expert annotated primary care physician notes from consecutive patients referred for cardiac testing without known atherosclerotic cardiovascular disease were included. Notes were annotated for positive and negative mentions of chest pain and shortness of breath …
Vicomtech Foundation（维孔特克基金会）；Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA））；HiTZ centre（HiTZ中心）；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））
This paper introduces the first version of the NUBes corpus (Negation and Uncertainty annotations in Biomedical texts in Spanish). The corpus is part of an on-going research and currently consists of 29,682 sentences obtained from anonymised health records annotated with negation and uncertainty. The article includes an exhaustive comparison with similar corpora in Spanish, and presents the main annotation and design decisions. Additionally, we perform preliminary experiments using deep learning algorithms to validate the annotated dataset. As far as we know, NUBes is the largest publicly avai…
Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (EMNLP'18) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qual…
University of Exeter（埃克塞特大学）；University of Pisa（比萨大学）；ISTI-CNR（意大利国家研究委员会信息科学与技术研究所）
Writing messages is key to expressing feelings. This study adopts cognitive network science to reconstruct how individuals report their feelings in clinical narratives like suicide notes or mental health posts. We achieve this by reconstructing syntactic/semantic associations between conceptsin texts as co-occurrences enriched with affective data. We transform 142 suicide notes and 77,000 Reddit posts from the r/anxiety, r/depression, r/schizophrenia, and r/do-it-your-own (r/DIY) forums into 5 cognitive networks, each one expressing meanings and emotions as reported by authors. These networks …
Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare
Stanford University（斯坦福大学）；University of Colorado（科罗拉多大学）；Northwestern University（西北大学）；University of Wisconsin（威斯康星大学）；Yale School of Medicine（耶鲁医学院）；University of Chicago（芝加哥大学）；Ohio State University（俄亥俄州立大学）
HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
Medical information extraction consists of a group of natural language processing (NLP) tasks, which collaboratively convert clinical text to pre-defined structured formats. Current state-of-the-art (SOTA) NLP models are highly integrated with deep learning techniques and thus require massive annotated linguistic data. This study presents an engineering framework of medical entity recognition, relation extraction and attribute extraction, which are unified in annotation, modeling and evaluation. Specifically, the annotation scheme is comprehensive, and compatible between tasks, especially for …
Population age information is an essential characteristic of clinical trials. In this paper, we focus on extracting minimum and maximum (min/max) age values for the study samples from clinical research articles. Specifically, we investigate the use of a neural network model for question answering to address this information extraction task. The min/max age QA model is trained on the massive structured clinical study records from ClinicalTrials.gov. For each article, based on multiple min and max age values extracted from the QA model, we predict both actual min/max age values for the study sam…
Reykjavik University（雷克雅未克大学）；Primary Health Care Service of the Capital Area（首都地区初级卫生保健服务中心）；University of Iceland（冰岛大学）；Development Centre for Primary Health Care in Iceland（冰岛初级卫生保健发展中心）
Clinical Text Notes (CTNs) contain physicians' reasoning process, written in an unstructured free text format, as they examine and interview patients. In recent years, several studies have been published that provide evidence for the utility of machine learning for predicting doctors' diagnoses from CTNs, a task known as ICD coding. Data annotation is time consuming, particularly when a degree of specialization is needed, as is the case for medical data. This paper presents a method of augmenting a sparsely annotated dataset of Icelandic CTNs with a machine-learned imputation in a semi-self-su…
Keyword-optimized Template Insertion for Clinical Information Extraction via Prompt-based Learning
Mount Sinai（西奈山）；Hasso Plattner Institute for Digital Health at Mount Sinai（西奈山哈索·普拉特纳数字健康研究所）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Institute for Personalized Medicine（个性化医学研究所）
SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation
Department of Biostatistics and Bioinformatics, Duke University School of Medicine（杜克大学医学学院生物统计学与生物信息学系）；Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School（杜克-新加坡国立大学医学科学院AI+医学科学计划）；Centre for Biomedical Data Science, Duke-NUS Medical School（杜克-新加坡国立大学医学学院生物医学数据科学中心）；Department of Statistical Science, Duke University（杜克大学统计科学系）；Leiden University Medical Centre（莱顿大学医学中心）；Department of Mathematics, University of Texas at Austin（德克萨斯大学奥斯汀分校数学系）；Department of Internal Medicine, Yale School of Medicine（耶鲁医学院内科学系）；Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩尔曼医学院生物统计学、流行病学与信息学系）；The Graduate Group in Applied Mathematics and Computational Science, School of Arts and Sciences, University of Pennsylvania（宾夕法尼亚大学艺术与科学学院应用数学与计算科学联合组）；Medstar Health National Center for Human Factors in Healthcare, Washington, DC, USA（Medstar健康国家人因工程中心，华盛顿特区，美国）；Department of Biomedical Informatics, Columbia University（哥伦比亚大学生物医学信息学系）；Cancer Prevention and Control, Duke Cancer Institute, Durham, NC, USA（杜克癌症研究所癌症预防与控制部，达勒姆，北卡罗来纳州，美国）；Department of Population Health Sciences, Duke University School of Medicine（杜克大学医学学院流行病学与公共卫生系）；Division of Rheumatology and Immunology, Duke University School of Medicine（杜克大学医学学院风湿病学与免疫学系）；Pre-hospital and Emergency Research Centre, Health Services Research and Population Health, Duke-NUS Medical School（杜克-新加坡国立大学医学学院院前急救与应急研究中心，健康服务研究与人口健康）；NUS Artificial Intelligence Institute, National University of Singapore（新加坡国立大学人工智能研究所）；Division of Pulmonary, Allergy and Critical Care Medicine, Duke University School of Medicine（杜克大学医学学院呼吸科、过敏科与危重医学系）；Duke Center for Health Informatics, Duke University（杜克大学健康信息学中心）；Duke Clinical Research Institute, Durham, NC, USA（杜克临床研究中心，达勒姆，北卡罗来纳州，美国）
Leveraging deep active learning to identify low-resource mobility functioning information in public clinical notes
University of South Florida（南佛罗里达大学）；Moffitt Cancer Center（莫菲特癌症中心）；State University of New York at New Paltz（纽约州立大学新帕尔茨分校）；Oklahoma State University（俄克拉荷马州立大学）
Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting
Klaus Tschira Institute for Integrative Computational Cardiology（克劳斯·齐拉整合计算心脏病学研究所）；Heidelberg University（海德堡大学）；University Hospital Heidelberg（海德堡大学医院）；German Center for Cardiovascular Research - Partner site Heidelberg/Mannheim（德国心血管研究中心海德堡/曼海姆合作站点）；Informatics for Life（生命信息学机构）
Institute for Infocomm Research (I2R), A*STAR（新加坡科技研究局资讯通信研究院）
There is growing interest in the automated extraction of relevant information from clinical dialogues. However, it is difficult to collect and construct large annotated resources for clinical dialogue tasks. Recent developments in natural language processing suggest that large-scale pre-trained language backbones could be leveraged for such machine comprehension and information extraction tasks. Yet, due to the gap between pre-training and downstream clinical domains, it remains challenging to exploit the generic backbones for domain-specific applications. Therefore, in this work, we propose a…
Dental-TriageBench: Benchmarking Multimodal Reasoning for Hierarchical Dental Triage
School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）；Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；The Prince Philip Dental Hospital（菲利普王子牙科医院）；Li Ka Shing Faculty of Medicine, The University of Hong Kong（香港大学利滋医学学院）；The Hong Kong University of Science and Technology（香港科学与技术大学）
