Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting quality should focus more on matching the disease keywords and their associated attributes instead of counting the occurrence of N-gram. Based on these concerns, we propose to utilize a pre-constructed graph embedding module (modeled with a graph convolutional neural network) on multiple disease findings to assist the generation of reports in this work. The incorporation of knowledge graph allows for dedicated feature learning for each disease finding and the relationship modeling between them. In addition, we proposed a new evaluation metric for radiology image reporting with the assistance of the same composed graph. Experimental results demonstrate the superior performance of the methods integrated with the proposed graph embedding module on a publicly accessible dataset (IU-RR) of chest radiographs compared with previous approaches using both the conventional evaluation metrics commonly adopted for image captioning and our proposed ones.
Although deep learning models for chest X-ray interpretation are commonly trained on labels generated by automatic radiology report labelers, the impact of improvements in report labeling on the performance of chest X-ray classification models has not been systematically investigated. We first compare the CheXpert, CheXbert, and VisualCheXbert labelers on the task of extracting accurate chest X-ray image labels from radiology reports, reporting that the VisualCheXbert labeler outperforms the CheXpert and CheXbert labelers. Next, after training image classification models using labels generated…
Automatic extraction of medical conditions from free-text radiology reports is critical for supervising computer vision models to interpret medical images. In this work, we show that radiologists labeling reports significantly disagree with radiologists labeling corresponding chest X-ray images, which reduces the quality of report labels as proxies for image labels. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images. Our best performing method, called VisualCheXbert, uses a biomedically-pretrained BERT model to …
University of California San Diego（加州大学圣迭戈分校）；Birla Institute of Technology and Science, Pilani（比尔拉科学技术学院皮拉尼分校）；Carnegie Mellon University（卡内基梅隆大学）
To effectively train medical students to become qualified radiologists, a large number of X-ray images collected from patients with diverse medical conditions are needed. However, due to data privacy concerns, such images are typically difficult to obtain. To address this problem, we develop methods to generate view-consistent, high-fidelity, and high-resolution X-ray images from radiology reports to facilitate radiology training of medical students. This task is presented with several challenges. First, from a single report, images with different views (e.g., frontal, lateral) need to be gene…
University of Chinese Academy of Sciences（中国科学院大学）；Tencent Medical AI Lab（腾讯医疗人工智能实验室）；University of Science and Technology of China（中国科学技术大学）
Automatic radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two…
Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Tsinghua University（清华大学）
Automated radiology report generation aims at automatically generating a detailed description of medical images, which can greatly alleviate the workload of radiologists and provide better medical services to remote areas. Most existing works pay attention to the holistic impression of medical images, failing to utilize important anatomy information. However, in actual clinical practice, radiologists usually locate important anatomical structures, and then look for signs of abnormalities in certain structures and reason the underlying disease. In this paper, we propose a novel framework AGFNet…
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；Weill Cornell Medicine（威尔康奈尔医学院）
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is ca…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Science and Technology of China（中国科学技术大学）；Tencent Medical AI Lab（腾讯医疗人工智能实验室）；University of Chinese Academy of Sciences（中国科学院大学）
In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical kn…
Acquiring high-quality annotations in medical imaging is usually a costly process. Automatic label extraction with natural language processing (NLP) has emerged as a promising workaround to bypass the need of expert annotation. Despite the convenience, the limitation of such an approximation has not been carefully examined and is not well understood. With a challenging set of 1,000 chest X-ray studies and their corresponding radiology reports, we show that there exists a surprisingly large discrepancy between what radiologists visually perceive and what they clinically report. Furthermore, wit…
Semantically Informed Salient Regions Guided Radiology Report Generation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；Department of Radiology, Peking University Third Hospital（北京大学第三医院放射科）
Radiology Report Generation Using Transformers Conditioned with Non-imaging Data
Centre for Computational Imaging and Simulation Technologies in Biomedicine（生物医学计算成像与模拟技术中心）；Leeds Institute for Cardiovascular and Metabolic Medicine（利兹心血管与代谢医学研究所）；School of Medicine（医学院）；School of Computing, University of Leeds（利兹大学计算机学院）
本文提出一种多模态 Transformer 网络，整合胸部 X 射线图像与患者人口统计信息生成放射报告，实验证明加入非影像数据有效提升了生成质量。
University of Rochester（罗切斯特大学）；Futurewei Technologies, Inc.（未来 Wei 技术公司）
Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limite…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Duke University（杜克大学）；The Chinese University of Hong Kong（香港中文大学）；Brigham and Women’s Hospital, Harvard Medical School（布莱根妇女医院，哈佛医学院）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Medical imaging plays a pivotal role in diagnosis and treatment in clinical practice. Inspired by the significant progress in automatic image captioning, various deep learning (DL)-based methods have been proposed to generate radiology reports for medical images. Despite promising results, previous works overlook the uncertainties of their models and are thus unable to provide clinicians with the reliability/confidence of the generated radiology reports to assist their decision-making. In this paper, we propose a novel method to explicitly quantify both the visual uncertainty and the textual u…
John Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Italian Institute of Technology（意大利技术研究院）；University of California, San Francisco（旧金山大学）；University of California, Berkeley（伯克利大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
University of Toronto（多伦多大学）；The Hospital for Sick Children (SickKids)（病童医院（SickKids））；Vector Institute（向量研究所）
Traditional datasets for the radiological diagnosis tend to only provide the radiology image alongside the radiology report. However, radiology reading as performed by radiologists is a complex process, and information such as the radiologist's eye-fixations over the course of the reading has the potential to be an invaluable data source to learn from. Nonetheless, the collection of such data is expensive and time-consuming. This leads to the question of whether such data is worth the investment to collect. This paper utilizes the recently published Eye-Gaze dataset to perform an exhaustive st…
Massachusetts Institute of Technology（麻省理工学院）；Philips Research North America（飞利浦北美研究院）；Beth Israel Deaconess Medical Center, Harvard Medical School（哈佛医学院贝斯以色列女执事医疗中心）
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiog…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Augnito India Pvt Ltd（印度Augnito私人有限公司）
Conventionally, the radiologist prepares the diagnosis notes and shares them with the transcriptionist. Then the transcriptionist prepares a preliminary formatted report referring to the notes, and finally, the radiologist reviews the report, corrects the errors, and signs off. This workflow causes significant delays and errors in the report. In current research work, we focus on applications of NLP techniques like Information Extraction (IE) and domain-specific Knowledge Graph (KG) to automatically generate radiology reports from radiologist's dictation. This paper focuses on KG construction …
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Weill Cornell Medicine（威尔康奈尔医学院）
Radiology report generation aims to produce computer-aided diagnoses to alleviate the workload of radiologists and has drawn increasing attention recently. However, previous deep learning methods tend to neglect the mutual influences between medical findings, which can be the bottleneck that limits the quality of generated reports. In this work, we propose to mine and represent the associations among medical findings in an informative knowledge graph and incorporate this prior knowledge with radiology report generation to help improve the quality of generated reports. Experiment results demons…
Radiology report generation (RRG) aims to describe automatically a radiology image with human-like language and could potentially support the work of radiologists, reducing the burden of manual reporting. Previous approaches often adopt an encoder-decoder architecture and focus on single-modal feature learning, while few studies explore cross-modal feature interaction. Here we propose a Cross-modal PROtotype driven NETwork (XPRONET) to promote cross-modal pattern learning and exploit it to improve the task of radiology report generation. This is achieved by three well-designed, fully different…
Philips Algotec, a Philips Co.（飞利浦阿尔戈泰克公司（飞利浦旗下公司））；Royal Philips（皇家飞利浦公司）
An infrastructure for multisite, geographically-distributed creation and collection of diverse, high-quality, curated and labeled radiology image data is crucial for the successful automated development, deployment, monitoring and continuous improvement of Artificial Intelligence (AI)/Machine Learning (ML) solutions in the real world. An interactive radiology reporting approach that integrates image viewing, dictation, natural language processing (NLP) and creation of hyperlinks between image findings and the report, provides localized labels during routine interpretation. These images and lab…
MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models
Microsoft Health Futures（微软健康未来）；Microsoft Research India（微软研究院印度分院）；University of Cambridge（剑桥大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院国民保健信托基金）
CBM-RAG: Demonstrating Enhanced Interpretability in Radiology Report Generation with Multi-Agent RAG and Concept Bottleneck Models
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Saarland University（萨尔兰大学）；University of Oldenburg（奥尔登堡大学）；Dhaka Medical College Hospital（达卡医学院医院）
School of Biomedical Engineering, King’s College London（伦敦国王学院生物医学工程学院）；King’s College Hospital（国王学院医院）；Institute of Psychiatry, Psychology & Neuroscience, King’s College London（伦敦国王学院精神病学、心理学与神经科学研究所）；Centre for Medical Image Computing, Dementia Research, University College London（伦敦大学学院医学影像计算中心（痴呆研究））
Labelling large datasets for training high-capacity neural networks is a major obstacle to the development of deep learning-based medical imaging applications. Here we present a transformer-based network for magnetic resonance imaging (MRI) radiology report classification which automates this task by assigning image labels on the basis of free-text expert radiology reports. Our model's performance is comparable to that of an expert radiologist, and better than that of an expert physician, demonstrating the feasibility of this approach. We make code available online for researchers to label the…
IBM Research（IBM研究公司）；IBM Research San Jose, CA, USA（美国加利福尼亚州圣何塞IBM研究公司（注：按要求取到公司级别，此处“IBM Research”已明确为公司，无需重复提取））
Data labeling is currently a time-consuming task that often requires expert knowledge. In research settings, the availability of correctly labeled data is crucial to ensure that model predictions are accurate and useful. We propose relatively simple machine learning-based models that achieve high performance metrics in the binary and multiclass classification of radiology reports. We compare the performance of these algorithms to that of a data-driven approach based on NLP, and find that the logistic regression classifier outperforms all other models, in both the binary and multiclass classifi…
CCS: Clinical Consensus Selection for Radiology Report Generation
School of Computing Science, University of Glasgow（格拉斯哥大学计算机科学学院）；School of Electrical and Computer Engineering, University of Sydney（悉尼大学电气与计算机工程学院）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
