Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learning
The University of Texas Health Science Center（德克萨斯大学健康科学中心）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）
Surabhi Datta, Yuqi Si, Laritza Rodriguez, Sonya E Shooshan, Dina Demner-Fushman, Kirk Roberts
We define a representation framework for extracting spatial information from radiology reports (Rad-SpRL). We annotated a total of 2000 chest X-ray reports with 4 spatial roles corresponding to the common radiology entities. Our focus is on extracting detailed information of a radiologist's interpretation containing a radiographic finding, its anatomical location, corresponding probable diagnoses, as well as associated hedging terms. For this, we propose a deep learning-based natural language processing (NLP) method involving both word and character-level encodings. Specifically, we utilize a bidirectional long short-term memory (Bi-LSTM) conditional random field (CRF) model for extracting the spatial roles. The model achieved average F1 measures of 90.28 and 94.61 for extracting the Trajector and Landmark roles respectively whereas the performance was moderate for Diagnosis and Hedge roles with average F1 of 71.47 and 73.27 respectively. The corpus will soon be made available upon request.
Extraction of relevant pathological terms from radiology reports is important for correct image label generation and disease population studies. In this letter, we compare the performance of some known application program interface (APIs) for the task of thoracic abnormality extraction from radiology reports. We explored several medical domain specific annotation tools like Medical Text Indexer(MTI) with Non-MEDLINE and Mesh On Demand(MOD) options and generic Natural Language Understanding (NLU) API provided by the IBM cloud. Our results show that although MTI and MOD are intended for extracti…
Extracting structured clinical information from free-text radiology reports can enable the use of radiology report information for a variety of critical healthcare applications. In our work, we present RadGraph, a dataset of entities and relations in full-text chest X-ray radiology reports based on a novel information extraction schema we designed to structure radiology reports. We release a development dataset, which contains board-certified radiologist annotations for 500 radiology reports from the MIMIC-CXR dataset (14,579 entities and 10,889 relations), and a test dataset, which contains t…
Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. These systems have achieved promising performance as measured by widely used NLG metrics such as BLEU and CIDEr. However, the current systems face important limitations. First, they present an increased complexity in architecture that offers only marginal improvements on NLG metrics. Secondly, these systems that achieve high performance on these metrics are not always factually complete or cons…
German CheXpert Chest X-ray Radiology Report Labeler
Munich Institute of Biomedical Engineering, Technical University of Munich（慕尼黑工业大学慕尼黑生物医学工程研究所）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；University Hospital, LMU Munich（慕尼黑大学附属医院）；Institute of Neuroradiology, University Hospital, LMU Munich（慕尼黑大学附属医院神经放射学研究所）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Radiology report generation aims at generating descriptive text from radiology images automatically, which may present an opportunity to improve radiology reporting and interpretation. A typical setting consists of training encoder-decoder models on image-report pairs with a cross entropy loss, which struggles to generate informative sentences for clinical diagnoses since normal findings dominate the datasets. To tackle this challenge and encourage more clinically-accurate text outputs, we propose a novel weakly supervised contrastive loss for medical report generation. Experimental results de…
Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning
Munich Institute of Biomedical Engineering, Technical University of Munich（慕尼黑工业大学慕尼黑生物医学工程研究所）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；University Hospital, LMU Munich（慕尼黑大学附属医院）；Institute of Neuroradiology, University Hospital, LMU Munich（慕尼黑大学附属医院神经放射学研究所）
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；Weill Cornell Medicine（威尔康奈尔医学院）
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is ca…
Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports
National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆国家生物技术信息中心）；National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）；Weill Cornell Medicine（威尔康奈尔医学院）
University of Rochester（罗切斯特大学）；Ludwig Maximilian University（路德维希-马克西米利安大学）
Deep learning models can be applied successfully in real-work problems; however, training most of these models requires massive data. Recent methods use language and vision, but unfortunately, they rely on datasets that are not usually publicly available. Here we pave the way for further research in the multimodal language-vision domain for radiology. In this paper, we train a representation learning method that uses local and global representations of the language and vision through an attention mechanism and based on the publicly available Indiana University Radiology Report (IU-RR) dataset.…
University of Toronto（多伦多大学）；Massachusetts Institute of Technology（麻省理工学院）；Vector Institute（矢量研究所）
The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation s…
Temple University（天普大学）；University Clinical Center of Serbia（塞尔维亚大学临床中心）
Beyond their primary diagnostic purpose, radiology reports have been an invaluable source of information in medical research. Given a corpus of radiology reports, researchers are often interested in identifying a subset of reports describing a particular medical finding. Because the space of medical findings in radiology reports is vast and potentially unlimited, recent studies proposed mapping free-text statements in radiology reports to semi-structured strings of terms taken from a limited vocabulary. This paper aims to present an approach for the automatic generation of semi-structured repr…
PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation
Microsoft Research（微软研究院）；Medbravo；Department of Radiology, University of Cambridge（剑桥大学放射科）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基础研究信托）；University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院）；Department of Health Informatics, University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院健康信息学系）；University Institute for Computing Research, University of Alicante（阿利坎特大学计算研究学院）
Harvard University（哈佛大学）；Stanford University（斯坦福大学）；Harvard Medical School（哈佛医学院）
Current deep learning models trained to generate radiology reports from chest radiographs are capable of producing clinically accurate, clear, and actionable text that can advance patient care. However, such systems all succumb to the same problem: making hallucinated references to non-existent prior reports. Such hallucinations occur because these models are trained on datasets of real-world patient reports that inherently refer to priors. To this end, we propose two methods to remove references to priors in radiology reports: (1) a GPT-3-based few-shot approach to rewrite medical reports wit…
Stanford University（斯坦福大学）；Harvard University（哈佛大学）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）；University of Texas Health Science Center at San Antonio（德克萨斯大学圣安东尼奥健康科学中心）；Jawaharlal Institute of Postgraduate Medical Education and Research（贾瓦哈拉尔研究生医学教育与研究学院）
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Stanford University（斯坦福大学）；University of Oxford（牛津大学）；University Hospital Zurich, University of Zurich（苏黎世大学医院、苏黎世大学）；Yongin Severance Hospital, Yonsei University（永临松医院、延世大学）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Google（谷歌）
Automatic medical image report generation has drawn growing attention due to its potential to alleviate radiologists' workload. Existing work on report generation often trains encoder-decoder networks to generate complete reports. However, such models are affected by data bias (e.g.~label imbalance) and face common issues inherent in text generation models (e.g.~repetition). In this work, we focus on reporting abnormal findings on radiology images; instead of training on complete radiology reports, we propose a method to identify abnormal findings from the reports in addition to grouping them …
Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation
KAIST（韩国科学技术院）；Ghent University（根特大学）；Seoul National University Hospital（首尔国立大学医院）；Seoul Medical Center（首尔医院）；Yeungnam University College of Medicine（延世大学医学院）；Microsoft Research Health Futures（微软研究院健康未来）
Improving the Performance of Radiology Report De-identification with Large-Scale Training and Benchmarking Against Cloud Vendor Methods
Stanford University（斯坦福大学）；JP Morgan Chase & Co（摩根大通公司）；Sorbonne University（索邦大学）；University of Oxford（牛津大学）；NVIDIA（英伟达）；HOPPR；University of Pennsylvania（宾夕法尼亚大学）
The extraction of labels from radiology text reports enables large-scale training of medical imaging models. Existing approaches to report labeling typically rely either on sophisticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT-based approach to medical image report labeling that exploits both the scale of available rule-based systems and the quality of expert annotations. We demonstrate superior performance of a biomedically pretrained BERT model first trained on annotations of a rule-based labeler and then finet…
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo（东京大学医学研究生院放射学与生物医学工程系）；Department of Radiology, The University of Tokyo Hospital（东京大学医院放射科）；Department of Radiology, Kanazawa University Graduate School of Medical Sciences（金泽大学医学研究生院放射科）；Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital（东京大学医院计算诊断放射学与预防医学系）；Department of Radiology, School of Medicine, Jichi Medical University（自治医科大学医学院放射科）
Radiology reports contain a diverse and rich set of clinical abnormalities documented by radiologists during their interpretation of the images. Comprehensive semantic representations of radiological findings would enable a wide range of secondary use applications to support diagnosis, triage, outcomes prediction, and clinical research. In this paper, we present a new corpus of radiology reports annotated with clinical findings. Our annotation schema captures detailed representations of pathologic findings that are observable on imaging ("lesions") and other types of clinical problems ("medica…
Acquiring high-quality annotations in medical imaging is usually a costly process. Automatic label extraction with natural language processing (NLP) has emerged as a promising workaround to bypass the need of expert annotation. Despite the convenience, the limitation of such an approximation has not been carefully examined and is not well understood. With a challenging set of 1,000 chest X-ray studies and their corresponding radiology reports, we show that there exists a surprisingly large discrepancy between what radiologists visually perceive and what they clinically report. Furthermore, wit…
KU-DMIS-MSRA at RadSum23: Pre-trained Vision-Language Model for Radiology Report Summarization
Korea University（高丽大学）；Microsoft Research Asia（微软亚洲研究院）；KAIST AI（韩国科学技术院人工智能学院）；Beihang University（北京航空航天大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）；AIGEN Sciences Inc.（AIGEN科学公司）
提出面向胸部 X 光的预训练视觉-语言模型 CheXOFA，通过多任务统一、跨领域迁移、集成与事实校准提升放射学报告摘要性能，并在 RadSum23 隐藏测试集取得第一名。
Clinical Context-aware Radiology Report Generation from Medical Images using Transformers
School of Computer Science and Engineering（计算机科学与工程学院）；Faculty of Engineering, University of New South Wales（新南威尔士大学工程学院）；School of Computing（计算机学院）；Faculty of Science and Engineering, Macquarie University（麦考瑞大学科学与工程学院）
本研究提出基于 transformer 的架构从胸部 X 光片生成放射学报告，实验证明其优于 LSTM 且速度更快，并强调需结合语言生成与分类指标以全面评估报告的连贯性与诊断价值。
RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation
Department of Computer Science, University of California, Davis, CA, USA（加州大学戴维斯分校计算机科学系）；Department of Radiology, University of California, Davis, CA, USA（加州大学戴维斯分校放射学系）
MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models
Microsoft Health Futures（微软健康未来）；Microsoft Research India（微软研究院印度分院）；University of Cambridge（剑桥大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院国民保健信托基金）
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Science and Technology of China（中国科学技术大学）；Tencent Medical AI Lab（腾讯医疗人工智能实验室）；University of Chinese Academy of Sciences（中国科学院大学）
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical kn…
PRECISE Framework: GPT-based Text For Improved Readability, Reliability, and Understandability of Radiology Reports For Patient-Centered Care
Massachusetts General Hospital（麻省总医院）；Brigham and Women's Hospital（布列根和妇女医院）；Athinoula A. Martinos Center for Biomedical Imaging（阿西诺拉·A·马蒂诺斯生物医学成像中心）；Harvard Medical School（哈佛医学院）；Julia R. Masterman High School（朱莉娅·R·马斯特曼高中）
