大语言模型（LLMs）的致命缺陷是幻觉问题，这在临床领域会造成严重后果，尤其是在自动生成出院小结（总结住院就诊情况的长篇医疗文档）的场景下影响尤为突出。自动生成这类小结可让医生腾出时间照料患者，减轻文书工作负担。本研究旨在探索结合语言类图结构与深度学习模型的新方法，解决自动摘要中的内容溯源与可信度问题。我们的方法在公开的Medical Information Mart for Intensive III（MIMIC-III，重症监护医疗信息库III）语料库，以及匿名医院医生撰写的临床笔记上，取得了优异的可靠性结果。我们公开了所提方法、生成的出院小结输出示例、源代码与训练好的模型。
英文摘要：
The Achilles heel of Large Language Models (LLMs) is hallucination, which has drastic consequences for the clinical domain. This is particularly important with regards to automatically generating discharge summaries (a lengthy medical document that summarizes a hospital in-patient visit). Automatically generating these summaries would free physicians to care for patients and reduce documentation burden. The goal of this work is to discover new methods that combine language-based graphs and deep learning models to address provenance of content and trustworthiness in automatic summarization. Our method shows impressive reliability results on the publicly available Medical Information Mart for Intensive III (MIMIC-III) corpus and clinical notes written by physicians at Anonymous Hospital. rovide our method, generated discharge ary output examples, source code and trained models.
Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital
Department of Health Information Office, Data & Digitalization, University Medical Center（健康信息办公室、数据与数字化化大学医院）；Department of Trauma Surgery / Orthopedics, University Medical Center（创伤外科/骨科大学医院）；Department of Orthopaedic Surgery, Massachusetts General Hospital（骨科手术科马萨诸塞总医院）；Harvard Medical School Orthopaedic Trauma Initiative（哈佛医学院骨创伤倡议）；Department of Orthopaedic Surgery and Sports Medicine, Amsterdam UMC location University of Amsterdam（骨科手术与运动医学部门，阿姆斯特丹UMC地点大学）；Department of Orthopaedic Trauma, Flinders University Medical Centre（骨创伤部门，弗林德斯大学医疗中心）；Department of Otolaryngology – Head and Neck Surgery, University Medical Center（耳鼻喉科-头颈外科大学医院）；Department of Pediatrics, Pediatrics Nephrology, Beatrix Children’s Hospital, University Medical Center（儿科、儿科肾病学，比西特儿童医院，大学医院）
Summarization of clinical narratives is a long-standing research problem. Here, we introduce the task of hospital-course summarization. Given the documentation authored throughout a patient's hospitalization, generate a paragraph that tells the story of the patient admission. We construct an English, text-to-text dataset of 109,000 hospitalizations (2M source notes) and their corresponding summary proxy: the clinician-authored "Brief Hospital Course" paragraph written as part of a discharge note. Exploratory analyses reveal that the BHC paragraphs are highly abstractive with some long extracte…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；School of Medicine and Public Health（医学与公共卫生学院）；Loyola University Chicago（芝加哥洛约拉大学）；Harvard Medical School（哈佛医学院）
Automatically summarizing patients' main problems from daily progress notes using natural language processing methods helps to battle against information and cognitive overload in hospital settings and potentially assists providers with computerized diagnostic decision support. Problem list summarization requires a model to understand, abstract, and generate clinical documentation. In this work, we propose a new NLP task that aims to generate a list of problems in a patient's daily care plan using input from the provider's progress notes during hospitalization. We investigate the performance o…
Tokyo Metropolitan University（东京都立大学）；National Hospital Organization（国立医院组织）；Kitami Institute of Technology（北见工业大学）；RIKEN（理化学研究所）
During the patient's hospitalization, the physician must record daily observations of the patient and summarize them into a brief document called "discharge summary" when the patient is discharged. Automated generation of discharge summary can greatly relieve the physicians' burden, and has been addressed recently in the research community. Most previous studies of discharge summary generation using the sequence-to-sequence architecture focus on only inpatient notes for input. However, electric health records (EHR) also have rich structured metadata (e.g., hospital, physician, disease, length …
Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models
King’s College London（伦敦国王学院）；King’s College Hospital NHS Foundation Trust（国王学院医院NHS基金会信托）；University College London（伦敦大学学院）
Columbia University（哥伦比亚大学）；Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心）
Long-form clinical summarization of hospital admissions has real-world significance because of its potential to help both clinicians and patients. The faithfulness of summaries is critical to their safe usage in clinical settings. To better understand the limitations of abstractive systems, as well as the suitability of existing evaluation metrics, we benchmark faithfulness metrics against fine-grained human annotations for model-generated summaries of a patient's Brief Hospital Course. We create a corpus of patient hospital admissions and summaries for a cohort of HIV patients, each with comp…
Patient summarization is essential for clinicians to provide coordinated care and practice effective communication. Automated summarization has the potential to save time, standardize notes, aid clinical decision making, and reduce medical errors. Here we provide an upper bound on extractive summarization of discharge notes and develop an LSTM model to sequentially label topics of history of present illness notes. We achieve an F1 score of 0.876, which indicates that this model can be employed to create a dataset for evaluation of extractive summarization methods.
WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV
University of Applied Sciences and Arts Dortmund（多特蒙德应用科学与艺术大学）；University Hospital Essen（埃森大学医院）；Bielefeld University of Applied Sciences and Arts（比勒费尔德应用科学与艺术大学）
University of Maryland（马里兰大学）；Amazon Web Service AI（亚马逊云服务人工智能）
The records of a clinical encounter can be extensive and complex, thus placing a premium on tools that can extract and summarize relevant information. This paper introduces the task of generating discharge summaries for a clinical encounter. Summaries in this setting need to be faithful, traceable, and scale to multiple long documents, motivating the use of extract-then-abstract summarization cascades. We introduce two new measures, faithfulness and hallucination rate for evaluation in this task, which complement existing measures for fluency and informativeness. Results across seven medical s…
University of New South Wales（新南威尔士大学）；The Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织澳大利亚电子健康研究中心）；Australian Institute of Health and Welfare（澳大利亚卫生与福利研究所）
Electronic Medical Records (EMRs) contain clinical narrative text that is of great potential value to medical researchers. However, this information is mixed with Personally Identifiable Information (PII) that presents risks to patient and clinician confidentiality. This paper presents an end-to-end deidentification framework to automatically remove PII from Australian hospital discharge summaries. Our corpus included 600 hospital discharge summaries which were extracted from the EMRs of two principal referral hospitals in Sydney, Australia. Our end-to-end de-identification framework consists …
School of Computer Science, The University of Manchester（曼彻斯特大学计算机学院）；Health eResearch Center, The University of Manchester（曼彻斯特大学健康电子研究中心）；The Alan Turing Institute（艾伦·图灵研究所）
Monitoring the administration of drugs and adverse drug reactions are key parts of pharmacovigilance. In this paper, we explore the extraction of drug mentions and drug-related information (reason for taking a drug, route, frequency, dosage, strength, form, duration, and adverse events) from hospital discharge summaries through deep learning that relies on various representations for clinical named entity recognition. This work was officially part of the 2018 n2c2 shared task, and we use the data supplied as part of the task. We developed two deep learning architecture based on recurrent neura…
Tokyo Metropolitan University（东京都立大学）；RIKEN（理化学研究所）；National Hospital Organization（国立医院机构）；Kitami Institute of Technology（北见工业大学）
Automated summarization of clinical texts can reduce the burden of medical professionals. "Discharge summaries" are one promising application of the summarization, because they can be generated from daily inpatient records. Our preliminary experiment suggests that 20-31% of the descriptions in discharge summaries overlap with the content of the inpatient records. However, it remains unclear how the summaries should be generated from the unstructured source. To decompose the physician's summarization process, this study aimed to identify the optimal granularity in summarization. We first define…
ASAPP；CUNY Graduate Center（纽约城市大学研究生中心）；United States Digital Service（美国数字服务局）；MIT（麻省理工学院）；Amazon Alexa AI（亚马逊Alexa AI）
Continuity of care is crucial to ensuring positive health outcomes for patients discharged from an inpatient hospital setting, and improved information sharing can help. To share information, caregivers write discharge notes containing action items to share with patients and their future caregivers, but these action items are easily lost due to the lengthiness of the documents. In this work, we describe our creation of a dataset of clinical action items annotated over MIMIC-III, the largest publicly available dataset of real clinical notes. This dataset, which we call CLIP, is annotated by phy…
MIMIC-IV-Ext-PE: Using a large language model to predict pulmonary embolism phenotype in the MIMIC-IV dataset
Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Baylor College of Medicine（贝勒医学院）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；Massachusetts Institute of Technology（麻省理工学院）
Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine
University of Oxford（牛津大学）；Peking University（北京大学）；Yale University（耶鲁大学）；Tencent（腾讯）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits
Department of Computer Science and Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad Campus（计算机科学与信息系统系，比拉理工学院，比拉理工学院，海得拉巴校区）
Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；Princeton University（普林斯顿大学）；New York University（纽约大学）
Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intens…
Beuth University of Applied Sciences Berlin（柏林比乌特应用科学大学）；Charité Berlin（柏林夏里特医学院）
Outcome prediction from clinical text can prevent doctors from overlooking possible risks and help hospitals to plan capacities. We simulate patients at admission time, when decision support can be especially valuable, and contribute a novel admission to discharge task with four common outcome prediction targets: Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction. The ideal system should infer outcomes based on symptoms, pre-conditions and risk factors of a patient. We evaluate the effectiveness of language models to handle this scenario and propo…
Northeast Yucai Foreign Language School（东北育才外国语学校）；Northeast Yucai School（东北育才学校）；Shenyang Pharmaceutical University（沈阳药科大学）
ICU readmission is associated with longer hospitalization, mortality and adverse outcomes. An early recognition of ICU re-admission can help prevent patients from worse situation and lower treatment cost. As the abundance of Electronics Health Records (EHR), it is popular to design clinical decision tools with machine learning technique manipulating on healthcare large scale data. We designed data-driven predictive models to estimate the risk of ICU readmission. The discharge summary of each hospital admission was carefully represented by natural language processing techniques. Unified Medical…
Stanford University（斯坦福大学）；University of Florida（佛罗里达大学）
Neural abstractive summarization models are able to generate summaries which have high overlap with human references. However, existing models are not optimized for factual correctness, a critical metric in real-world applications. In this work, we develop a general framework where we evaluate the factual correctness of a generated summary by fact-checking it automatically against its reference using an information extraction module. We further propose a training strategy which optimizes a neural summarization model with a factual correctness reward via reinforcement learning. We apply the pro…
Columbia University（哥伦比亚大学）；Amazon AWS AI（亚马逊AWS人工智能部门）
In real-world scenarios with naturally occurring datasets, reference summaries are noisy and may contain information that cannot be inferred from the source text. On large news corpora, removing low quality samples has been shown to reduce model hallucinations. Yet, for smaller, and/or noisier corpora, filtering is detrimental to performance. To improve reference quality while retaining all data, we propose a new approach: to selectively re-write unsupported reference sentences to better reflect source data. We automatically generate a synthetic dataset of positive and negative revisions by co…
Hainan University（海南大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
This study presents three deidentified large medical text datasets, named DISCHARGE, ECHO and RADIOLOGY, which contain 50K, 16K and 378K pairs of report and summary that are derived from MIMIC-III, respectively. We implement convincing baselines of automated abstractive summarization on the proposed datasets with pre-trained encoder-decoder language models, including BERT2BERT, T5-large and BART. Further, based on the BART model, we leverage the sampled summaries from the train set as prior knowledge guidance, for encoding additional contextual representations of the guidance with the encoder …
Imperial College London（伦敦帝国理工学院）；Transformative AI（变革性人工智能公司）
Past studies on the ICD coding problem focus on predicting clinical codes primarily based on the discharge summary. This covers only a small fraction of the notes generated during each hospital stay and leaves potential for improving performance by analysing all the available clinical notes. We propose a hierarchical transformer architecture that uses text across the entire sequence of clinical notes in each hospital stay for ICD coding, and incorporates embeddings for text metadata such as their position, time, and type of note. While using all clinical notes increases the quantity of data su…
Australian e-Health Research Centre, CSIRO（澳大利亚联邦科学与工业研究组织电子健康研究中心）；Royal Brisbane and Women’s Hospital（皇家布里斯班妇女医院）
Reviewing radiology reports in emergency departments is an essential but laborious task. Timely follow-up of patients with abnormal cases in their radiology reports may dramatically affect the patient's outcome, especially if they have been discharged with a different initial diagnosis. Machine learning approaches have been devised to expedite the process and detect the cases that demand instant follow up. However, these approaches require a large amount of labeled data to train reliable predictive models. Preparing such a large dataset, which needs to be manually annotated by health professio…
The development of neural networks for clinical artificial intelligence (AI) is reliant on interpretability, transparency, and performance. The need to delve into the black-box neural network and derive interpretable explanations of model output is paramount. A task of high clinical importance is predicting the likelihood of a patient being readmitted to hospital in the near future to enable efficient triage. With the increasing adoption of electronic health records (EHRs), there is great interest in applications of natural language processing (NLP) to clinical free-text contained within EHRs.…
Unplanned intensive care unit (ICU) readmission rate is an important metric for evaluating the quality of hospital care. Efficient and accurate prediction of ICU readmission risk can not only help prevent patients from inappropriate discharge and potential dangers, but also reduce associated costs of healthcare. In this paper, we propose a new method that uses medical text of Electronic Health Records (EHRs) for prediction, which provides an alternative perspective to previous studies that heavily depend on numerical and time-series features of patients. More specifically, we extract discharge…
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
KAIST（韩国科学技术院）；Seoul National University（首尔大学）；Seoul National University Bundang Hospital（首尔大学盆唐医院）；SAIHST, Sungkyunkwan University（成均馆大学）；Yonsei University College of Medicine（延世大学医学院）；Gangnam Severance Hospital（江南塞弗伦斯医院）；Severance Hospital（塞弗伦斯医院）；Seoul Medical Center（首尔医疗中心）；Seoul National University Hospital（首尔大学医院）；National Cancer Center（国立癌症中心）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Samsung Medical Center（三星医疗中心）
A language model can be used to predict the next word during authoring, to correct spelling or to accelerate writing (e.g., in sms or emails). Language models, however, have only been applied in a very small scale to assist physicians during authoring (e.g., discharge summaries or radiology reports). But along with the assistance to the physician, computer-based systems which expedite the patient's exit also assist in decreasing the hospital infections. We employed statistical and neural language modeling to predict the next word of a clinical text and assess all the models in terms of accurac…
Unsupervised pretraining is an integral part of many natural language processing systems, and transfer learning with language models has achieved remarkable results in many downstream tasks. In the clinical application of medical code assignment, diagnosis and procedure codes are inferred from lengthy clinical notes such as hospital discharge summaries. However, it is not clear if pretrained models are useful for medical code prediction without further architecture engineering. This paper conducts a comprehensive quantitative analysis of various contextualized language models' performance, pre…
