Objective: To comparatively evaluate several transformer model architectures at identifying protected health information (PHI) in the i2b2/UTHealth 2014 clinical text de-identification challenge corpus.
Methods: The i2b2/UTHealth 2014 corpus contains N=1304 clinical notes obtained from N=296 patients. Using a transfer learning framework, we fine-tune several transformer model architectures on the corpus, including: BERT-base, BERT-large, ROBERTA-base, ROBERTA-large, ALBERT-base and ALBERT-xxlarge. During fine-tuning we vary the following model hyper-parameters: batch size, number training epochs, learning rate and weight decay. We fine tune models on a training data set, we evaluate and select optimally performing models on an independent validation dataset, and lastly assess generalization performance on a held-out test dataset. We assess model performance in terms of accuracy, precision (positive predictive value), recall (sensitivity) and F1 score (harmonic mean of precision and recall). We are interested in overall model performance (PHI identified vs. PHI not identified), as well as PHI-specific model performance.
Results: We observe that the ROBERTA-large models perform best at identifying PHI in the i2b2/UTHealth 2014 corpus, achieving >99% overall accuracy and 96.7% recall/precision on the heldout test corpus. Performance was good across many PHI classes; however, accuracy/precision/recall decreased for identification of the following entity classes: professions, organizations, ages, and certain locations.
Conclusions: Transformers are a promising model class/architecture for clinical text de-identification. With minimal hyper-parameter tuning transformers afford researchers/clinicians the opportunity to obtain (near) state-of-the-art performance.
Improving Extraction of Clinical Event Contextual Properties from Electronic Health Records: A Comparative Study
King’s College London(伦敦国王学院);Guy’s and St Thomas’ NHS Foundation Trust(盖伊和圣托马斯 NHS 基金会信托);King’s College Hospital NHS Foundation Trust(国王学院医院 NHS 基金会信托);University College London(伦敦大学学院);Health Data Research UK London(英国健康数据研究伦敦中心)
DeIDClinic: A Risk-Aware Pseudonymization Framework for Clinical Text De-identification and Re-identification Risk Assessment
University of Manchester(曼彻斯特大学);Leiden Institute of Advanced Computer Science (LIACS)(莱顿先进计算机科学研究所);Leiden University(莱顿大学);Biomedical Data Sciences, Leiden University Medical Center(莱顿大学医学中心生物医学数据科学)
Evaluating Large Language Models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025:A Comparative Analysis of Clinical Reasoning and Knowledge Application
Reliable Automated Triage in Spanish Clinical Notes: A Hybrid Framework for Risk-Aware HIV Suspicion Identification
Dept. of Lenguajes y Sistemas Informáticos, Escuela Técnica Superior de Ingeniería Informática, Universidad Nacional de Educación a Distancia (UNED)(语言与系统信息学系,信息工程技术学校,国家远程教育大学(UNED));Dept. Informática y Estadística, Escuela Técnica Superior de Ingeniería Informática, Universidad Rey Juan Carlos (URJC)(信息与统计学系,信息工程技术学校,皇家胡安·卡洛斯大学(URJC))
Not What the Doctor Ordered: Surveying LLM-based De-identification and Quantifying Clinical Information Loss
Department of Computing Science, University of Alberta(计算科学系,阿尔伯塔大学);Department of Medicine, University of Alberta(医学系,阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所);Arthritis Research Canada(关节炎研究加拿大)
De-identification is not enough: a comparison between de-identified and synthetic clinical notes
Department of Computer Science, University of Manitoba(曼尼托巴大学计算机科学系);McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心麦克威廉姆斯生物医学信息学院)
Comparing representations of long clinical texts for the task of patient note-identification
Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);Data Science Platform, Imagine Institute(伊马金研究所数据科学平台);Hôpital Universitaire Necker-Enfants Malades(内克尔-恩范茨-马拉德大学医院);Assistance Publique - Hôpitaux de Paris (APHP)(巴黎公共医疗援助医院集团);Université de Lorraine(洛林大学);CNRS(法国国家科学研究中心);LORIA(洛林计算机科学与应用实验室);INESC-ID(里斯本大学学院计算机与系统工程研究中心);Instituto Superior Técnico(里斯本高等技术学院);Universidade de Lisboa(里斯本大学)
Extracting Patient History from Clinical Text: A Comparative Study of Clinical Large Language Models
Oklahoma State University(俄克拉荷马州立大学);University of South Florida(南佛罗里达大学);South Dakota School of Mines and Technology(南达科他矿业理工学院);University of South Florida Morsani College of Medicine(南佛罗里达大学莫萨尼医学院);South University School of Pharmacy(南大学药学院);The State University of New York Upstate Medical University(纽约州立大学上州医科大学);The State University of New York at New Paltz(纽约州立大学新帕尔茨分校)
Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnostic information from clinical CMR imaging reports
The D-Lab, Department of Precision Medicine, GROW - Research Institute for Oncology and Reproduction, Maastricht University(D实验室,精准医学系,GROW癌症与生殖研究研究所,马斯特里赫特大学);Division of Cardiovascular Imaging, Department of Cardiology I, University Hospital Münster(心血管影像科,心内科第一部门,穆恩斯特大学医院)
Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction
Center of Excellence in Research and Education for Big Military Data Intelligence (CREDIT Center)(大数据智能卓越研究中心(CREDIT中心));Department of Electrical and Computer Engineering(电气与计算机工程系);Prairie View A&M University(普拉提维大学);Texas A&M University System(德克萨斯A&M大学系统)
Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches
Department of Computing Science, University of Alberta(计算科学系,阿尔伯塔大学);Department of Medicine, University of Alberta(医学系,阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔人工智能研究所);Arthritis Research Canada(加拿大关节炎研究机构)
Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset
The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncology
Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ, United States(辐射肿瘤科,梅奥诊所,凤凰城,亚利桑那州,美国);Department of Radiation Oncology, Mayo Clinic, Rochester, MN, United States(辐射肿瘤科,梅奥诊所,罗切斯特,明尼苏达州,美国);Division of Radiation Oncology, IEO, European Institute of Oncology, IRCCS, Milan, Italy(放射肿瘤学部,IEO,欧洲肿瘤研究所,IRCCS,米兰,意大利)
DP-IPI: A Hybrid Differential Privacy Text Rewriting Mechanism for Indirect Personal Identifiers in Clinical Texts
Technical University of Berlin(柏林工业大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心)
Language-Specific versus Cross-Lingual Knowledge Graphs for Implicit Aspect Identification in Arabic: A Comparative Study of Reasoning and Adaptation Strategies