On the Effects of Low-Quality Training Data on Information Extraction from Clinical Reports
Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche（意大利国家研究委员会信息科学与技术研究所）；Qatar Computing Research Institute, Qatar Foundation（卡塔尔基金会卡塔尔计算研究所）；Consiglio Nazionale delle Ricerche（意大利国家研究委员会）
Diego Marcheggiani, Fabrizio Sebastiani
备注：
Submitted for publication
发表信息：
Final version published in ACM Journal of Data and Information Quality, 9(1), Article 1, 2017
In the last five years there has been a flurry of work on information extraction from clinical documents, i.e., on algorithms capable of extracting, from the informal and unstructured texts that are generated during everyday clinical practice, mentions of concepts relevant to such practice. Most of this literature is about methods based on supervised learning, i.e., methods for training an information extraction system from manually annotated examples. While a lot of work has been devoted to devising learning methods that generate more and more accurate information extractors, no work has been devoted to investigating the effect of the quality of training data on the learning process. Low quality in training data often derives from the fact that the person who has annotated the data is different from the one against whose judgment the automatically annotated data must be evaluated. In this paper we test the impact of such data quality issues on the accuracy of information extraction systems as applied to the clinical domain. We do this by comparing the accuracy deriving from training data annotated by the authoritative coder (i.e., the one who has also annotated the test data, and by whose judgment we must abide), with the accuracy deriving from training data annotated by a different coder. The results indicate that, although the disagreement between the two coders (as measured on the training set) is substantial, the difference is (surprisingly enough) not always statistically significant.
Use of 'off-the-shelf' information extraction algorithms in clinical informatics: a feasibility study of MetaMap annotation of Italian medical notes
Consiglio Nazionale delle Ricerche (CNR)（意大利国家研究委员会）；Politecnico di Milano（米兰理工大学）；Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico Milano（米兰大医院基金会）
Efficient extraction of medication information from clinical notes: an evaluation in two languages
University hospital of Strasbourg（斯特拉斯堡大学医院）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；Université Paris Cité（巴黎西岱大学）；HeKA；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；LISN（巴黎第十一大学计算机科学实验室）
A Hybrid Citation Retrieval Algorithm for Evidence-based Clinical Knowledge Summarization: Combining Concept Extraction, Vector Similarity and Query Expansion for High Precision
Northwestern University Feinberg School of Medicine（西北大学范伯格医学院）；Bluhm Cardiovascular Institute of Northwestern Medicine（西北大学布鲁姆心血管研究所）；University of Kansas Medical Center（堪萨斯大学医学中心）
Keyword-optimized Template Insertion for Clinical Information Extraction via Prompt-based Learning
Mount Sinai（西奈山）；Hasso Plattner Institute for Digital Health at Mount Sinai（西奈山哈索·普拉特纳数字健康研究所）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Institute for Personalized Medicine（个性化医学研究所）
A Unified Framework of Medical Information Annotation and Extraction for Chinese Clinical Text
HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
A Medical Information Extraction Workbench to Process German Clinical Text
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Charité – Universitätsmedizin Berlin（柏林夏里特医学院）；Berlin Institute of Health（柏林健康研究所）
Analysing zero-shot temporal relation extraction on clinical notes using temporal consistency
Faculty of Computer Science, University of Vienna（维也纳大学计算机科学学院）；UniVie Doctoral School Computer Science（维也纳大学计算机博士学院）；Aarhus University（奥胡斯大学）；Faculty of Philological and Cultural Studies, University of Vienna（维也纳大学语言与文化学院）
Clinical QA 2.0: Multi-Task Learning for Answer Extraction and Categorization
University of Washington（华盛顿大学）；New York University（纽约大学）；Liverpool John Moores University（利物浦约翰·摩尔斯大学）；Birla Institute of Technology Pilani（比拉理工学院帕利尼分校）；Columbia University（哥伦比亚大学）
Enhancing Clinical Information Extraction with Transferred Contextual Embeddings
The Australian National University (ANU)（澳大利亚国立大学）；Data61, Commonwealth Scientific and Industrial Research Organization (CSIRO)（澳大利亚联邦科学与工业研究组织Data61部门）；University of Canberra（堪培拉大学）；University of Turku（图尔库大学）
Distilling Large Language Models for Efficient Clinical Information Extraction
Poolesville High School（普尔斯维尔高中）；University of California Santa Cruz（加州大学圣克鲁兹分校）；Stanford University School of Medicine（斯坦福大学医学院）；Clinical Excellence Research Center（临床卓越研究中心）；Stanford Health Care（斯坦福医疗保健系统）
Assessment of Amazon Comprehend Medical: Medication Information Extraction
NYU Langone Health（纽约大学朗格尼医学中心）
2018年11月27日，Amazon Web Services（AWS）发布了Amazon Comprehend Medical（ACM），这是一个基于深度学习的系统，可以从临床文本笔记中自动提取临床概念（包括解剖结构、医疗状况、受保护的健康信息（PH）I、测试名称、治疗名称、医疗程序和药物）。对任何新数据产品的接受和信任都依赖于跨基准数据集和工具的独立验证，以建立和确认预期的结果质量。这项工作的重点是药物提取任务，特别是使用2009年i2b2药物提取挑战赛和2018年n2c2轨道2的官方测试集评估ACM：不良药物事件和EHR中的药物提取。总体而言，ACM的F评分为0.768和0.828。与各自挑战中的三个最佳系统相比，这些分数排名最低。为了进一步确定其药物提取性能的普遍性，这项工作还包括来自纽约大学Langone医学中心的一组随机内部临床文本注释。在这个语料库中，ACM获得了0.753的F分数。
Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?
McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心麦克威廉姆斯生物医学信息学学院）；Yale School of Medicine, Yale University（耶鲁大学医学院）
SIFTING: A Novel LLM-Based Framework for Structured and Transparent Information Extraction from Clinical Free-Text Reports, with Application to Tumor Staging in Lung Cancer
Axana B.V.（阿克萨纳有限公司）；University College London Hospitals NHS Foundation Trust（伦敦大学学院医院 NHS 基金会信托）；University College London（伦敦大学学院）；Mid & South Essex NHS Foundation Trust（中埃塞克斯及南埃塞克斯 NHS 基金会信托）；NHS Greater Glasgow and Clyde（大格拉斯哥及克莱德 NHS）
Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction
McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston（德克萨斯大学健康科学中心休斯顿分校麦克威廉斯生物医学信息学学院）；School of Public Health, The University of Texas Health Science Center at Houston（德克萨斯大学健康科学中心休斯顿分校公共卫生学院）；School of Dentistry, The University of Texas Health Science Center at Houston（德克萨斯大学健康科学中心休斯顿分校牙科学院）；Willamette Dental and Skourtes Institute（威廉特牙科与斯库尔特斯研究所）
Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（健康结果与生物医学信息学系，医学院，佛罗里达大学）；Division of Pulmonary, Critical Care and Sleep Medicine, Department of Medicine, College of Medicine, University of Florida（呼吸科、重症医学科和睡眠医学科，医学系，医学院，佛罗里达大学）；College of Nursing, Florida State University（护理学院，佛罗里达州立大学）
