Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development
IBM Research（IBM研究）；Almaden Research Center（阿尔马登研究中心）；Department of Radiology and Imaging Sciences, Emory University（埃默里大学放射与影像科学系）；Emory University（埃默里大学）
Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A Kr… 展开作者
Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A Krupinski, Mehdi Moradi
We developed a rich dataset of Chest X-Ray (CXR) images to assist investigators in artificial intelligence. The data were collected using an eye tracking system while a radiologist reviewed and reported on 1,083 CXR images. The dataset contains the following aligned data: CXR image, transcribed radiology report text, radiologist's dictation audio and eye gaze coordinates data. We hope this dataset can contribute to various areas of research particularly towards explainable and multimodal deep learning / machine learning methods. Furthermore, investigators in disease classification and localization, automated radiology report generation, and human-machine interaction can benefit from these data. We report deep learning experiments that utilize the attention maps produced by eye gaze dataset to show the potential utility of this data.
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与影像研究所）；Department of Radiology and Imaging Sciences, University of Utah（犹他大学放射学与影像科学系）
Convolutional neural networks (CNNs) have been successfully applied to chest x-ray (CXR) images. Moreover, annotated bounding boxes have been shown to improve the interpretability of a CNN in terms of localizing abnormalities. However, only a few relatively small CXR datasets containing bounding boxes are available, and collecting them is very costly. Opportunely, eye-tracking (ET) data can be collected in a non-intrusive way during the clinical workflow of a radiologist. We use ET data recorded from radiologists while dictating CXR reports to train CNNs. We extract snippets from the ET data b…
University of Utah（犹他大学）；Scientific Computing and Imaging Institute（科学计算与成像研究所）
Deep learning has shown recent success in classifying anomalies in chest x-rays, but datasets are still small compared to natural image datasets. Supervision of abnormality localization has been shown to improve trained models, partially compensating for dataset sizes. However, explicitly labeling these anomalies requires an expert and is very time-consuming. We propose a potentially scalable method for collecting implicit localization data using an eye tracker to capture gaze locations and a microphone to capture a dictation of a report, imitating the setup of a reading room. The resulting RE…
INESC-ID（INESC-ID（葡萄牙国家系统与计算机工程研究所））；Instituto Superior Técnico（里斯本高等技术学院）；Universidade de Lisboa（里斯本大学）；School of Information Systems（信息系统学院）；Queensland University of Technology（昆士兰科技大学）；Grupo Lusíadas（卢西亚达斯集团）
This paper proposes a novel multimodal DL architecture incorporating medical images and eye-tracking data for abnormality detection in chest x-rays. Our results show that applying eye gaze data directly into DL architectures does not show superior predictive performance in abnormality detection chest X-rays. These results support other works in the literature and suggest that human-generated data, such as eye gaze, needs a more thorough investigation before being applied to DL architectures.
University of Amsterdam（阿姆斯特丹大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Medical image datasets and their annotations are not growing as fast as their equivalents in the general domain. This makes translation from the newest, more data-intensive methods that have made a large impact on the vision field increasingly more difficult and less efficient. In this paper, we propose a new probabilistic latent variable model for disease classification in chest X-ray images. Specifically we consider chest X-ray datasets that contain global disease labels, and for a smaller subset contain object level expert annotations in the form of eye gaze patterns and disease bounding bo…
Harvard University（哈佛大学）；Institute for Applied Computational Science（应用计算科学研究所）
Heart failure remains a major public health challenge with growing costs. Ejection fraction (EF) is a key metric for the diagnosis and management of heart failure however estimation of EF using echocardiography remains expensive for the healthcare system and subject to intra/inter operator variability. While chest x-rays (CXR) are quick, inexpensive, and require less expertise, they do not provide sufficient information to the human eye to estimate EF. This work explores the efficacy of computer vision techniques to predict reduced EF solely from CXRs. We studied a dataset of 3488 CXRs from th…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In this era of pandemic, the future of healthcare industry has never been more exciting. Artificial intelligence and machine learning (AI & ML) present opportunities to develop solutions that cater for very specific needs within the industry. Deep learning in healthcare had become incredibly powerful for supporting clinics and in transforming patient care in general. Deep learning is increasingly being applied for the detection of clinically important features in the images beyond what can be perceived by the naked human eye. Chest X-ray images are one of the most common clinical method for di…
The increased availability and accuracy of eye-gaze tracking technology has sparked attention-related research in psychology, neuroscience, and, more recently, computer vision and artificial intelligence. The attention mechanism in artificial neural networks is known to improve learning tasks. However, no previous research has combined the network attention and human attention. This paper describes a gaze-guided class activation mapping (GG-CAM) method to directly regulate the formation of network attention based on expert radiologists' visual attention for the chest X-ray pathology classifica…
GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs
Department of Biomedical Informatics, Stony Brook University, NY, US（斯通比克大学生物医学信息学系）；Department of Radiology, Columbia University, NY, US（哥伦比亚大学放射学系）；Department of Computer Science, Stony Brook University, NY, US（斯通比克大学计算机科学系）
Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study
The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Northwestern University（西北大学）；University of Texas Health Science Center at San Antonio（德克萨斯大学圣安东尼奥健康科学中心）；Long School of Medicine（朗医学院）
Comparing radiologists' gaze and saliency maps generated by interpretability methods for chest x-rays
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；School of Computing, University of Utah（犹他大学计算学院）；Department of Psychology, University of Utah（犹他大学心理学系）；Department of Radiology and Imaging Sciences, University of Utah（犹他大学放射学与影像科学系）
GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph
AI Research Center for Medical Image Analysis and Diagnosis, Shenzhen University（深圳大学医学影像分析与诊断AI研究中心）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院计算机视觉研究所）；Department of Electrical Engineering, City University of Hong Kong（香港城市大学电气工程系）
Mask of truth: model sensitivity to unexpected regions of medical images
IT University of Copenhagen（丹麦哥本哈根技术大学）；Copenhagen University Hospital, Herlev and Gentofte（哥本哈根大学医院，赫尔勒夫和琴托夫）；Radiological AI Testcenter（放射学AI测试中心）；Department of Radiology, Zealand University Hospital（放射科， Zealand大学医院）
In this work, we present RadioTransformer, a novel visual attention-driven transformer framework, that leverages radiologists' gaze patterns and models their visuo-cognitive behavior for disease diagnosis on chest radiographs. Domain experts, such as radiologists, rely on visual information for medical image interpretation. On the other hand, deep neural networks have demonstrated significant promise in similar tasks even where visual interpretation is challenging. Eye-gaze tracking has been used to capture the viewing behavior of domain experts, lending insights into the complexity of visual …
A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；Kahlert School of Computing, University of Utah（犹他大学卡勒特计算学院）；University of Utah（犹他大学）
University of Toronto（多伦多大学）；The Hospital for Sick Children (SickKids)（病童医院（SickKids））；Vector Institute（向量研究所）
As deep learning is widely used in the radiology field, the explainability of such models is increasingly becoming essential to gain clinicians' trust when using the models for diagnosis. In this research, three experiment sets were conducted with a U-Net architecture to improve the classification performance while enhancing the heatmaps corresponding to the model's focus through incorporating heatmap generators during training. All of the experiments used the dataset that contained chest radiographs, associated labels from one of the three conditions ("normal", "congestive heart failure (CHF)…
Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis
College of Computer Science, Northwest University（西北大学计算机学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Department of PET/CT, The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院PET/CT科）
Qatar University（卡塔尔大学）；Universiti Kebangsaan Malaysia（马来西亚国民大学）
Novel Coronavirus disease (COVID-19) is an extremely contagious and quickly spreading Coronavirus infestation. Severe Acute Respiratory Syndrome (SARS) and Middle East Respiratory Syndrome (MERS), which outbreak in 2002 and 2011, and the current COVID-19 pandemic are all from the same family of coronavirus. This work aims to classify COVID-19, SARS, and MERS chest X-ray (CXR) images using deep Convolutional Neural Networks (CNNs). A unique database was created, so-called QU-COVID-family, consisting of 423 COVID-19, 144 MERS, and 134 SARS CXR images. Besides, a robust COVID-19 recognition syste…
A Clinical-oriented Multi-level Contrastive Learning Method for Disease Diagnosis in Low-quality Medical Images
Northeastern University（东北大学）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education（医学图像智能计算教育部重点实验室）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization（工业智能与系统优化全国前沿科学中心）
MDF-Net for abnormality detection by fusing X-rays with clinical data
Queensland University of Technology（昆士兰科技大学）；Lusíadas Knowledge Center（卢西亚达斯知识中心）；Instituto de Sistemas e Robótica（系统与机器人研究所）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等技术学院）；INESC-ID；Data61 CSIRO
Shadow defense against gradient inversion attack in federated learning
Bioengineering Department, Imperial College London, London W12 0BZ, UK；School of Computer Engineering；Science, Shanghai university, Shanghai 200444, China；Bioengineering Department；Imperial-X, Imperial College London, London W12 7SL, UK；Lung Institute, Imperial College London, London SW7 2AZ, UK；Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, UK；School of Biomedical Engineering \& Imaging Sciences, King's College London, London WC2R 2LS, UK
