Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR
Although deep learning models for chest X-ray interpretation are commonly trained on labels generated by automatic radiology report labelers, the impact of improvements in report labeling on the performance of chest X-ray classification models has not been systematically investigated. We first compare the CheXpert, CheXbert, and VisualCheXbert labelers on the task of extracting accurate chest X-ray image labels from radiology reports, reporting that the VisualCheXbert labeler outperforms the CheXpert and CheXbert labelers. Next, after training image classification models using labels generated…
Massachusetts Institute of Technology（麻省理工学院）；Philips Research North America（飞利浦北美研究院）；Beth Israel Deaconess Medical Center, Harvard Medical School（哈佛医学院贝斯以色列女执事医疗中心）
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiog…
International Institute of Information Technology（国际信息技术研究所）
Deep learning (DL) analysis of Chest X-ray (CXR) and Computed tomography (CT) images has garnered a lot of attention in recent times due to the COVID-19 pandemic. Convolutional Neural Networks (CNNs) are well suited for the image analysis tasks when trained on humongous amounts of data. Applications developed for medical image analysis require high sensitivity and precision compared to any other fields. Most of the tools proposed for detection of COVID-19 claims to have high sensitivity and recalls but have failed to generalize and perform when tested on unseen datasets. This encouraged us to …
Automatic extraction of medical conditions from free-text radiology reports is critical for supervising computer vision models to interpret medical images. In this work, we show that radiologists labeling reports significantly disagree with radiologists labeling corresponding chest X-ray images, which reduces the quality of report labels as proxies for image labels. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images. Our best performing method, called VisualCheXbert, uses a biomedically-pretrained BERT model to …
Radiology reports are an important means of communication between radiologists and other physicians. These reports express a radiologist's interpretation of a medical imaging examination and are critical in establishing a diagnosis and formulating a treatment plan. In this paper, we propose a Bi-directional convolutional neural network (Bi-CNN) model for the interpretation and classification of mammograms based on breast density and chest radiographic radiology reports based on the basis of chest pathology. The proposed approach helps to organize databases of radiology reports, retrieve them e…
University of Rochester（罗切斯特大学）；Ludwig Maximilian University（路德维希-马克西米利安大学）
Deep learning models can be applied successfully in real-work problems; however, training most of these models requires massive data. Recent methods use language and vision, but unfortunately, they rely on datasets that are not usually publicly available. Here we pave the way for further research in the multimodal language-vision domain for radiology. In this paper, we train a representation learning method that uses local and global representations of the language and vision through an attention mechanism and based on the publicly available Indiana University Radiology Report (IU-RR) dataset.…
PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation
Microsoft Research（微软研究院）；Medbravo；Department of Radiology, University of Cambridge（剑桥大学放射科）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基础研究信托）；University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院）；Department of Health Informatics, University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院健康信息学系）；University Institute for Computing Research, University of Alicante（阿利坎特大学计算研究学院）
We propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country. To assess the efficacy of the proposed labeling technique, we built a CXR dataset containing 9,752 studies and evaluated our pipeline using a subset of this dataset. With an F1-score of at least 0.9923, the evaluation demonstrates that our labeling tool perform…
University of Toronto（多伦多大学）；Massachusetts Institute of Technology（麻省理工学院）；Vector Institute（矢量研究所）
The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation s…
AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via $α$-Corrected Binary Cross Entropy and Factorized Latent Supervision
Peter Munk Cardiac Centre, University Health Network (UHN)（彼得·芒克心脏中心，大学健康网络（UHN））；Ted Rogers Centre for Heart Research（泰德·罗杰斯心脏研究中心）；University of Toronto (U of T)（多伦多大学）；Vector Institute（向量研究所）；Toronto General Hospital Research Institute, UHN（多伦多总医院研究所，大学健康网络（UHN））
Stanford University（斯坦福大学）；Harvard University（哈佛大学）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）；University of Texas Health Science Center at San Antonio（德克萨斯大学圣安东尼奥健康科学中心）；Jawaharlal Institute of Postgraduate Medical Education and Research（贾瓦哈拉尔研究生医学教育与研究学院）
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Department of Computer Science and Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Koita Centre for Digital Health, Indian Institute of Technology Bombay（印度理工学院孟买分校Koita数字健康中心）
University of Rochester（罗切斯特大学）；Futurewei Technologies, Inc.（未来 Wei 技术公司）
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limite…
Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports
Rensselaer Polytechnic Institute, NY, USA（罗文学院）；IBM Research, Almaden, CA, USA（IBM研究院）；Stanford University, CA, USA（斯坦福大学）；Massachusetts General Hospital (MGH), Boston, USA（麻省总医院）
Acquiring high-quality annotations in medical imaging is usually a costly process. Automatic label extraction with natural language processing (NLP) has emerged as a promising workaround to bypass the need of expert annotation. Despite the convenience, the limitation of such an approximation has not been carefully examined and is not well understood. With a challenging set of 1,000 chest X-ray studies and their corresponding radiology reports, we show that there exists a surprisingly large discrepancy between what radiologists visually perceive and what they clinically report. Furthermore, wit…
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Stanford University（斯坦福大学）；University of Oxford（牛津大学）；University Hospital Zurich, University of Zurich（苏黎世大学医院、苏黎世大学）；Yongin Severance Hospital, Yonsei University（永临松医院、延世大学）
IBM Research（IBM研究公司）；IBM Research San Jose, CA, USA（美国加利福尼亚州圣何塞IBM研究公司（注：按要求取到公司级别，此处“IBM Research”已明确为公司，无需重复提取））
Data labeling is currently a time-consuming task that often requires expert knowledge. In research settings, the availability of correctly labeled data is crucial to ensure that model predictions are accurate and useful. We propose relatively simple machine learning-based models that achieve high performance metrics in the binary and multiclass classification of radiology reports. We compare the performance of these algorithms to that of a data-driven approach based on NLP, and find that the logistic regression classifier outperforms all other models, in both the binary and multiclass classifi…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Google（谷歌）
Automatic medical image report generation has drawn growing attention due to its potential to alleviate radiologists' workload. Existing work on report generation often trains encoder-decoder networks to generate complete reports. However, such models are affected by data bias (e.g.~label imbalance) and face common issues inherent in text generation models (e.g.~repetition). In this work, we focus on reporting abnormal findings on radiology images; instead of training on complete radiology reports, we propose a method to identify abnormal findings from the reports in addition to grouping them …
Machine learning models for radiology benefit from large-scale data sets with high quality labels for abnormalities. We curated and analyzed a chest computed tomography (CT) data set of 36,316 volumes from 19,993 unique patients. This is the largest multiply-annotated volumetric medical imaging data set reported. To annotate this data set, we developed a rule-based method for automatically extracting abnormality labels from free-text radiology reports with an average F-score of 0.976 (min 0.941, max 1.0). We also developed a model for multi-organ, multi-disease classification of chest CT volum…
IBM Research（IBM研究）；Almaden Research Center（阿尔马登研究中心）；Department of Radiology and Imaging Sciences, Emory University（埃默里大学放射与影像科学系）；Emory University（埃默里大学）
We developed a rich dataset of Chest X-Ray (CXR) images to assist investigators in artificial intelligence. The data were collected using an eye tracking system while a radiologist reviewed and reported on 1,083 CXR images. The dataset contains the following aligned data: CXR image, transcribed radiology report text, radiologist's dictation audio and eye gaze coordinates data. We hope this dataset can contribute to various areas of research particularly towards explainable and multimodal deep learning / machine learning methods. Furthermore, investigators in disease classification and localiza…
Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting qua…
Christian Brothers University（基督教兄弟大学）；University of Dayton（代顿大学）
With a Coronavirus disease (COVID-19) case count exceeding 10 million worldwide, there is an increased need for a diagnostic capability. The main variables in increasing diagnostic capability are reduced cost, turnaround or diagnosis time, and upfront equipment cost and accessibility. Two candidates for machine learning COVID-19 diagnosis are Computed Tomography (CT) scans and plain chest X-rays. While CT scans score higher in sensitivity, they have a higher cost, maintenance requirement, and turnaround time as compared to plain chest X-rays. The use of portable chest X-radiograph (CXR) is rec…
Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays
Universidade Federal de São Paulo (UNIFESP)（圣保罗联邦大学）；Hospital Israelita Albert Einstein（以色列阿尔伯特·爱因斯坦医院）；Stanford University School of Medicine（斯坦福大学医学院）；DASA；University of Texas Southwestern Medical Center (UTSW)（德克萨斯大学西南医学中心）；Eden
Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation
CISTIB Centre for Computational Imaging and Simulation Technologies in Biomedicine（生物医学计算成像与模拟技术中心（CISTIB））；School of Computing, University of Leeds（利兹大学计算机学院）；School of Languages, Cultures and Societies, University of Leeds（利兹大学语言、文化与社会学院）；Kastamonu Training and Research Hospital（卡斯塔莫努培训与研究医院）；School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester（曼彻斯特大学生物、医学与健康学院健康科学学院）；School of Computer Science, Faculty of Science and Engineering, The University of Manchester（曼彻斯特大学科学与工程学院计算机学院）
Yale University（耶鲁大学）；Rice University（莱斯大学）；University of Texas Health Science Center（德克萨斯大学健康科学中心）
The proliferation of Deep Learning (DL)-based methods for radiographic image analysis has created a great demand for expert-labeled radiology data. Recent self-supervised frameworks have alleviated the need for expert labeling by obtaining supervision from associated radiology reports. These frameworks, however, struggle to distinguish the subtle differences between different pathologies in medical images. Additionally, many of them do not provide interpretation between image regions and text, making it difficult for radiologists to assess model predictions. In this work, we propose Local Regi…
Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）；Philips Research（飞利浦研究院）
Purpose: To develop a machine learning model to classify the severity grades of pulmonary edema on chest radiographs. Materials and Methods: In this retrospective study, 369,071 chest radiographs and associated radiology reports from 64,581 (mean age, 51.71; 54.51% women) patients from the MIMIC-CXR chest radiograph dataset were included. This dataset was split into patients with and without congestive heart failure (CHF). Pulmonary edema severity labels from the associated radiology reports were extracted from patients with CHF as four different ordinal levels: 0, no edema; 1, vascular conges…
Chest radiographs are among the most frequently acquired images in radiology and are often the subject of computer vision research. However, most of the models used to classify chest radiographs are derived from openly available deep neural networks, trained on large image-datasets. These datasets routinely differ from chest radiographs in that they are mostly color images and contain several possible image classes, while radiographs are greyscale images and often only contain fewer image classes. Therefore, very deep neural networks, which can represent more complex relationships in image-fea…
EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledge
Xidian University（西安电子科技大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Ruijin Hospital（瑞金医院）；Shanghai Institute of Traumatology and Orthopedics（上海市伤骨科研究所）；Harbin Medical University（哈尔滨医科大学）；The First Affiliated Hospital of Harbin Medical University（哈尔滨医科大学附属第一医院）；Brown University（布朗大学）
Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Xi’an Key Laboratory of Big Data and Intelligent Vision（西安大数据与智能视觉重点实验室）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University（教育部协同智能系统重点实验室）；Warren Alpert Medical School, Brown University（布朗大学沃伦·阿尔佩特医学学院）
Stanford AIMI（斯坦福大学医学人工智能实验室）；Stanford University（斯坦福大学）；University of California, Davis（加州大学戴维斯分校）；Stability AI
Multimodal models trained on large natural image-text pair datasets have exhibited astounding abilities in generating high-quality images. Medical imaging data is fundamentally different to natural images, and the language used to succinctly capture relevant details in medical data uses a different, narrow but semantically rich, domain-specific vocabulary. Not surprisingly, multi-modal models trained on natural image-text pairs do not tend to generalize well to the medical domain. Developing generative imaging models faithfully representing medical concepts while providing compositional divers…
Phu Tho Department of Health（富寿省卫生厅）；Vingroup Big Data Institute（Vingroup大数据研究院）；VinUniversity（VinUniversity大学）；Phu Tho General Hospital（富寿综合医院）
Computer-Aided Diagnosis (CAD) systems for chest radiographs using artificial intelligence (AI) have recently shown a great potential as a second opinion for radiologists. The performances of such systems, however, were mostly evaluated on a fixed dataset in a retrospective manner and, thus, far from the real performances in clinical practice. In this work, we demonstrate a mechanism for validating an AI-based system for detecting abnormalities on X-ray scans, VinDr-CXR, at the Phu Tho General Hospital - a provincial hospital in the North of Vietnam. The AI system was directly integrated into …
Massachusetts Institute of Technology（麻省理工学院）；Philips Research North America（飞利浦北美研究院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）
We propose and demonstrate machine learning algorithms to assess the severity of pulmonary edema in chest x-ray images of congestive heart failure patients. Accurate assessment of pulmonary edema in heart failure is critical when making treatment and disposition decisions. Our work is grounded in a large-scale clinical dataset of over 300,000 x-ray images with associated radiology reports. While edema severity labels can be extracted unambiguously from a small fraction of the radiology reports, accurate annotation is challenging in most cases. To take advantage of the unlabeled images, we deve…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Science and Technology of China（中国科学技术大学）；Tencent Medical AI Lab（腾讯医疗人工智能实验室）；University of Chinese Academy of Sciences（中国科学院大学）
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical kn…
