尽管在大规模数据上预训练的多模态基础模型已在自然语言理解和视觉识别领域取得成功，但由于医疗任务的细粒度特性以及对领域知识的高要求，其在医疗领域的应用仍十分有限。为应对这一挑战，我们提出了一种名为Knowledge-enhanced Auto Diagnosis（KAD，知识增强自动诊断）的新方法，该方法利用现有的医疗领域知识，结合配对的胸部X光片和放射报告来指导视觉语言预训练。我们在4个外部X光数据集上对KAD进行了评估，结果表明其零样本性能不仅与全监督模型相当，而且在5种病理中的3种上优于3名放射科专家的平均水平，且具有统计显著性。此外，在有少样本标注的情况下，KAD在微调设置下的表现优于所有现有方法，展现出其在不同临床场景中的应用潜力。
英文摘要：
While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nature of medical tasks and the high demand for domain knowledge. To address this challenge, we propose a novel approach called Knowledge-enhanced Auto Diagnosis (KAD) which leverages existing medical domain knowledge to guide vision-language pre-training using paired chest X-rays and radiology reports. We evaluate KAD on {four} external X-ray datasets and demonstrate that its zero-shot performance is not only comparable to that of fully-supervised models, but also superior to the average of three expert radiologists for three (out of five) pathologies with statistical significance. Moreover, when few-shot annotation is available, KAD outperforms all existing approaches in fine-tuning settings, demonstrating its potential for application in different clinical scenarios.
Massachusetts Institute of Technology（麻省理工学院）；Philips Research North America（飞利浦北美研究院）；Beth Israel Deaconess Medical Center, Harvard Medical School（哈佛医学院贝斯以色列女执事医疗中心）
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiog…
Radiology reports are an important means of communication between radiologists and other physicians. These reports express a radiologist's interpretation of a medical imaging examination and are critical in establishing a diagnosis and formulating a treatment plan. In this paper, we propose a Bi-directional convolutional neural network (Bi-CNN) model for the interpretation and classification of mammograms based on breast density and chest radiographic radiology reports based on the basis of chest pathology. The proposed approach helps to organize databases of radiology reports, retrieve them e…
Although deep learning models for chest X-ray interpretation are commonly trained on labels generated by automatic radiology report labelers, the impact of improvements in report labeling on the performance of chest X-ray classification models has not been systematically investigated. We first compare the CheXpert, CheXbert, and VisualCheXbert labelers on the task of extracting accurate chest X-ray image labels from radiology reports, reporting that the VisualCheXbert labeler outperforms the CheXpert and CheXbert labelers. Next, after training image classification models using labels generated…
AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via $α$-Corrected Binary Cross Entropy and Factorized Latent Supervision
Peter Munk Cardiac Centre, University Health Network (UHN)（彼得·芒克心脏中心，大学健康网络（UHN））；Ted Rogers Centre for Heart Research（泰德·罗杰斯心脏研究中心）；University of Toronto (U of T)（多伦多大学）；Vector Institute（向量研究所）；Toronto General Hospital Research Institute, UHN（多伦多总医院研究所，大学健康网络（UHN））
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Department of Computer Science and Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Koita Centre for Digital Health, Indian Institute of Technology Bombay（印度理工学院孟买分校Koita数字健康中心）
IBM Research（IBM研究）；Almaden Research Center（阿尔马登研究中心）；Department of Radiology and Imaging Sciences, Emory University（埃默里大学放射与影像科学系）；Emory University（埃默里大学）
We developed a rich dataset of Chest X-Ray (CXR) images to assist investigators in artificial intelligence. The data were collected using an eye tracking system while a radiologist reviewed and reported on 1,083 CXR images. The dataset contains the following aligned data: CXR image, transcribed radiology report text, radiologist's dictation audio and eye gaze coordinates data. We hope this dataset can contribute to various areas of research particularly towards explainable and multimodal deep learning / machine learning methods. Furthermore, investigators in disease classification and localiza…
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；Weill Cornell Medicine（威尔康奈尔医学院）
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is ca…
University of Rochester（罗切斯特大学）；Ludwig Maximilian University（路德维希-马克西米利安大学）
Deep learning models can be applied successfully in real-work problems; however, training most of these models requires massive data. Recent methods use language and vision, but unfortunately, they rely on datasets that are not usually publicly available. Here we pave the way for further research in the multimodal language-vision domain for radiology. In this paper, we train a representation learning method that uses local and global representations of the language and vision through an attention mechanism and based on the publicly available Indiana University Radiology Report (IU-RR) dataset.…
International Institute of Information Technology（国际信息技术研究所）
Deep learning (DL) analysis of Chest X-ray (CXR) and Computed tomography (CT) images has garnered a lot of attention in recent times due to the COVID-19 pandemic. Convolutional Neural Networks (CNNs) are well suited for the image analysis tasks when trained on humongous amounts of data. Applications developed for medical image analysis require high sensitivity and precision compared to any other fields. Most of the tools proposed for detection of COVID-19 claims to have high sensitivity and recalls but have failed to generalize and perform when tested on unseen datasets. This encouraged us to …
We propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country. To assess the efficacy of the proposed labeling technique, we built a CXR dataset containing 9,752 studies and evaluated our pipeline using a subset of this dataset. With an F1-score of at least 0.9923, the evaluation demonstrates that our labeling tool perform…
Massachusetts Institute of Technology（麻省理工学院）；Philips Research North America（飞利浦北美研究院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）
We propose and demonstrate machine learning algorithms to assess the severity of pulmonary edema in chest x-ray images of congestive heart failure patients. Accurate assessment of pulmonary edema in heart failure is critical when making treatment and disposition decisions. Our work is grounded in a large-scale clinical dataset of over 300,000 x-ray images with associated radiology reports. While edema severity labels can be extracted unambiguously from a small fraction of the radiology reports, accurate annotation is challenging in most cases. To take advantage of the unlabeled images, we deve…
University of Toronto（多伦多大学）；Massachusetts Institute of Technology（麻省理工学院）；Vector Institute（矢量研究所）
The automatic generation of radiology reports given medical radiographs has significant potential to operationally and improve clinical patient care. A number of prior works have focused on this problem, employing advanced methods from computer vision and natural language generation to produce readable reports. However, these works often fail to account for the particular nuances of the radiology domain, and, in particular, the critical importance of clinical accuracy in the resulting generated reports. In this work, we present a domain-aware automatic chest X-ray radiology report generation s…
Automatic extraction of medical conditions from free-text radiology reports is critical for supervising computer vision models to interpret medical images. In this work, we show that radiologists labeling reports significantly disagree with radiologists labeling corresponding chest X-ray images, which reduces the quality of report labels as proxies for image labels. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images. Our best performing method, called VisualCheXbert, uses a biomedically-pretrained BERT model to …
Deep Omni-supervised Learning for Rib Fracture Detection from Chest Radiology Images
Imsight AI Research Lab（Imsight AI 研究实验室）；The Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）；The Chinese University of Hong Kong（香港中文大学）
Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海自主智能无人系统科学中心）
A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluation
School of Electrical and Computer Engineering, The University of Sydney（电气与计算机工程学院，悉尼大学）；School of Computer Science, The University of Adelaide（计算机科学学院，阿德莱德大学）；School of Computing and Information Technology, University of Wollongong（计算与信息科技学院，沃伦冈大学）
PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation
Microsoft Research（微软研究院）；Medbravo；Department of Radiology, University of Cambridge（剑桥大学放射科）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基础研究信托）；University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院）；Department of Health Informatics, University Hospital Sant Joan d’Alacant（圣约翰·达·阿尔坎特医院健康信息学系）；University Institute for Computing Research, University of Alicante（阿利坎特大学计算研究学院）
Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports
Rensselaer Polytechnic Institute, NY, USA（罗文学院）；IBM Research, Almaden, CA, USA（IBM研究院）；Stanford University, CA, USA（斯坦福大学）；Massachusetts General Hospital (MGH), Boston, USA（麻省总医院）
University of Rochester（罗切斯特大学）；Futurewei Technologies, Inc.（未来 Wei 技术公司）
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limite…
Background: Chest X-rays are the most commonly performed, cost-effective diagnostic imaging tests ordered by physicians. A clinically validated AI system that can reliably separate normals from abnormals can be invaluble particularly in low-resource settings. The aim of this study was to develop and validate a deep learning system to detect various abnormalities seen on a chest X-ray. Methods: A deep learning system was trained on 2.3 million chest X-rays and their corresponding radiology reports to identify various abnormalities seen on a Chest X-ray. The system was tested against - 1. A thre…
Stanford University（斯坦福大学）；Harvard University（哈佛大学）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）；University of Texas Health Science Center at San Antonio（德克萨斯大学圣安东尼奥健康科学中心）；Jawaharlal Institute of Postgraduate Medical Education and Research（贾瓦哈拉尔研究生医学教育与研究学院）
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Stanford University（斯坦福大学）；University of Oxford（牛津大学）；University Hospital Zurich, University of Zurich（苏黎世大学医院、苏黎世大学）；Yongin Severance Hospital, Yonsei University（永临松医院、延世大学）
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Google（谷歌）
Automatic medical image report generation has drawn growing attention due to its potential to alleviate radiologists' workload. Existing work on report generation often trains encoder-decoder networks to generate complete reports. However, such models are affected by data bias (e.g.~label imbalance) and face common issues inherent in text generation models (e.g.~repetition). In this work, we focus on reporting abnormal findings on radiology images; instead of training on complete radiology reports, we propose a method to identify abnormal findings from the reports in addition to grouping them …
S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework
University of Sydney（悉尼大学）；University of Wollongong（沃林戈大学）；University of Adelaide（阿德莱德大学）；First Clinical Medical College, Guangzhou University of Chinese Medicine（广州中医药大学第一临床学院）
Obtaining automated preliminary read reports for common exams such as chest X-rays will expedite clinical workflows and improve operational efficiencies in hospitals. However, the quality of reports generated by current automated approaches is not yet clinically acceptable as they cannot ensure the correct detection of a broad spectrum of radiographic findings nor describe them accurately in terms of laterality, anatomical location, severity, etc. In this work, we present a domain-aware automatic chest X-ray radiology report generation algorithm that learns fine-grained description of findings…
Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays
Universidade Federal de São Paulo (UNIFESP)（圣保罗联邦大学）；Hospital Israelita Albert Einstein（以色列阿尔伯特·爱因斯坦医院）；Stanford University School of Medicine（斯坦福大学医学院）；DASA；University of Texas Southwestern Medical Center (UTSW)（德克萨斯大学西南医学中心）；Eden
IBM Research（IBM研究公司）；IBM Research San Jose, CA, USA（美国加利福尼亚州圣何塞IBM研究公司（注：按要求取到公司级别，此处“IBM Research”已明确为公司，无需重复提取））
Data labeling is currently a time-consuming task that often requires expert knowledge. In research settings, the availability of correctly labeled data is crucial to ensure that model predictions are accurate and useful. We propose relatively simple machine learning-based models that achieve high performance metrics in the binary and multiclass classification of radiology reports. We compare the performance of these algorithms to that of a data-driven approach based on NLP, and find that the logistic regression classifier outperforms all other models, in both the binary and multiclass classifi…
The potential of deep learning, especially in medical imaging, initiated astonishing results and improved the methodologies after every passing day. Deep learning in radiology provides the opportunity to classify, detect and segment different diseases automatically. In the proposed study, we worked on a non-trivial aspect of medical imaging where we classified and localized the X-Rays affected by bullets. We tested Images on different classification and localization models to get considerable accuracy. The replicated data set used in the study was replicated on different images of chest X-Rays…
Yale University（耶鲁大学）；Rice University（莱斯大学）；University of Texas Health Science Center（德克萨斯大学健康科学中心）
The proliferation of Deep Learning (DL)-based methods for radiographic image analysis has created a great demand for expert-labeled radiology data. Recent self-supervised frameworks have alleviated the need for expert labeling by obtaining supervision from associated radiology reports. These frameworks, however, struggle to distinguish the subtle differences between different pathologies in medical images. Additionally, many of them do not provide interpretation between image regions and text, making it difficult for radiologists to assess model predictions. In this work, we propose Local Regi…
Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）；Philips Research（飞利浦研究院）
Purpose: To develop a machine learning model to classify the severity grades of pulmonary edema on chest radiographs. Materials and Methods: In this retrospective study, 369,071 chest radiographs and associated radiology reports from 64,581 (mean age, 51.71; 54.51% women) patients from the MIMIC-CXR chest radiograph dataset were included. This dataset was split into patients with and without congestive heart failure (CHF). Pulmonary edema severity labels from the associated radiology reports were extracted from patients with CHF as four different ordinal levels: 0, no edema; 1, vascular conges…
A Utility-preserving De-identification Pipeline for Cross-hospital Radiology Data Sharing
Shandong University（山东大学）；Australian National University（澳大利亚国立大学）；Hong Kong University of Science and Technology（香港科技大学）；Curtin University（科廷大学）；Edith Cowan University（埃迪斯科文大学）；University of Western Australia（西澳大利亚大学）
An important component of human analysis of medical images and their context is the ability to relate newly seen things to related instances in our memory. In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis. By retrieving similar images and/or radiology reports we expand and regularize the case at hand with additional knowledge, while maintaining factual knowledge consistency. The method consists of two components. First, vision and language modalities are aligned using a pre-trained CLIP model. To enforce that …
EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledge
Xidian University（西安电子科技大学）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Ruijin Hospital（瑞金医院）；Shanghai Institute of Traumatology and Orthopedics（上海市伤骨科研究所）；Harbin Medical University（哈尔滨医科大学）；The First Affiliated Hospital of Harbin Medical University（哈尔滨医科大学附属第一医院）；Brown University（布朗大学）
Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Xi’an Key Laboratory of Big Data and Intelligent Vision（西安大数据与智能视觉重点实验室）；Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University（教育部协同智能系统重点实验室）；Warren Alpert Medical School, Brown University（布朗大学沃伦·阿尔佩特医学学院）
Phu Tho Department of Health（富寿省卫生厅）；Vingroup Big Data Institute（Vingroup大数据研究院）；VinUniversity（VinUniversity大学）；Phu Tho General Hospital（富寿综合医院）
Computer-Aided Diagnosis (CAD) systems for chest radiographs using artificial intelligence (AI) have recently shown a great potential as a second opinion for radiologists. The performances of such systems, however, were mostly evaluated on a fixed dataset in a retrospective manner and, thus, far from the real performances in clinical practice. In this work, we demonstrate a mechanism for validating an AI-based system for detecting abnormalities on X-ray scans, VinDr-CXR, at the Phu Tho General Hospital - a provincial hospital in the North of Vietnam. The AI system was directly integrated into …
