In this work, we compare the performance of six state-of-the-art deep neural networks in classification tasks when using only image features, to when these are combined with patient metadata. We utilise transfer learning from networks pretrained on ImageNet to extract image features from the ISIC HAM10000 dataset prior to classification. Using several classification performance metrics, we evaluate the effects of including metadata with the image features. Furthermore, we repeat our experiments with data augmentation. Our results show an overall enhancement in performance of each network as assessed by all metrics, only noting degradation in a vgg16 architecture. Our results indicate that this performance enhancement may be a general property of deep networks and should be explored in other areas. Moreover, these improvements come at a negligible additional cost in computation time, and therefore are a practical method for other applications.
The novel corona virus (Covid-19) has introduced significant challenges due to its rapid spreading nature through respiratory transmission. As a result, there is a huge demand for Artificial Intelligence (AI) based quick disease diagnosis methods as an alternative to high demand tests such as Polymerase Chain Reaction (PCR). Chest X-ray (CXR) Image analysis is such cost-effective radiography technique due to resource availability and quick screening. But, a sufficient and systematic data collection that is required by complex deep leaning (DL) models is more difficult and hence there are recen…
In this work we examine the performance enhancement in classification of medical imaging data when image features are combined with associated non-image data. We compare the performance of eight state-of-the-art deep neural networks in classification tasks when using only image features, compared to when these are combined with patient metadata. We utilise transfer learning with networks pretrained on ImageNet used directly as feature extractors and fine tuned on the target domain. Our experiments show that performance can be significantly enhanced with the inclusion of metadata and use interp…
Stanford University（斯坦福大学）；School of Medicine, Stanford University（斯坦福大学医学院）
Self-supervised contrastive learning between pairs of multiple views of the same image has been shown to successfully leverage unlabeled data to produce meaningful visual representations for both natural and medical images. However, there has been limited work on determining how to select pairs for medical images, where availability of patient metadata can be leveraged to improve representations. In this work, we develop a method to select positive pairs coming from views of possibly different images through the use of patient metadata. We compare strategies for selecting positive pairs for ch…
UT Austin（得克萨斯大学奥斯汀分校）；MIS, Germany（德国曼海姆应用科学大学）；Weill Cornell Medicine（威尔康奈尔医学院）
Computer-aided diagnosis plays a salient role in more accessible and accurate cardiopulmonary diseases classification and localization on chest radiography. Millions of people get affected and die due to these diseases without an accurate and timely diagnosis. Recently proposed contrastive learning heavily relies on data augmentation, especially positive data augmentation. However, generating clinically-accurate data augmentations for medical images is extremely difficult because the common data augmentation methods in computer vision, such as sharp, blur, and crop operations, can severely alt…
Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；The University of Queensland（昆士兰大学）；University of Athens Medical School（雅典大学医学院）；IBM Research（IBM研究院）；Universitat de Barcelona（巴塞罗那大学）；IDIBAPS（IDIBAPS（生物医学研究所））；Melanoma Institute Australia（澳大利亚黑色素瘤研究所）；Emory University School of Medicine（埃默里大学医学院）；Medical University of Vienna（维也纳医科大学）；Mayo Clinic（梅奥诊所）；SUNY Downstate Medical School（纽约州立大学唐斯泰特医学院）；Stony Brook Medical School（石溪大学医学院）；Rabin Medical Center（拉宾医疗中心）；Weill Cornell Medical College（威尔康奈尔医学院）
Prior skin image datasets have not addressed patient-level information obtained from multiple skin lesions from the same patient. Though artificial intelligence classification algorithms have achieved expert-level performance in controlled studies examining single images, in practice dermatologists base their judgment holistically from multiple lesions on the same patient. The 2020 SIIM-ISIC Melanoma Classification challenge dataset described herein was constructed to address this discrepancy between prior challenges and clinical practice, providing for each image in the dataset an identifier …
Metadata-enhanced contrastive learning from retinal optical coherence tomography images
Imperial College London（帝国理工学院）；Medical University of Vienna（维也纳医科大学）；University of Michigan（密歇根大学）；King’s College London（伦敦国王学院）；Institute of Molecular and Clinical Ophthalmology Basel（巴塞尔分子与临床眼科研究所）；Universitat Basel（巴塞尔大学）；University College London（伦敦大学学院）；Moorfields Eye Hospital（摩尔菲尔兹眼科医院）；University of Southampton（南安普顿大学）
Hybrid Ensemble of Segmentation-Assisted Classification and GBDT for Skin Cancer Detection with Engineered Metadata and Synthetic Lesions from ISIC 2024 Non-Dermoscopic 3D-TBP Images
Metadata-Aware Adaptation of a Generative Foundation Model for Conditional CMR Synthesis
Facultat de Matemàtiques i Informàtica, Universitat de Barcelona（巴塞罗那大学数学与信息学院）；NIHR Barts Biomedical Research Centre（英国国立卫生研究院巴特生物医学研究中心）；St Bartholomew’s Hospital（圣巴塞洛缪医院）；Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚高级研究学院）
Combining Microscopy Data and Metadata for Reconstruction of Cellular Traction Forces Using a Hybrid Vision Transformer-U-Net
Fakultät Physik, Technische Universität Dortmund（物理系，德意志理工大学多特蒙德分校）；Faculty of Physics and Center for NanoScience, Ludwig-Maximilians-Universität München（物理系和纳米科学中心，路德维希-马克西米利安大学慕尼黑分校）；Department of Veterinary Sciences, Ludwig-Maximilians-Universität München（兽医学系，路德维希-马克西米利安大学慕尼黑分校）
Polytechnique Montreal（蒙特利尔理工学院）；Mila, Quebec AI Institute（魁北克人工智能研究所米拉）；Beijing Tiantan Hospital, Capital Medical University（首都医科大学附属北京天坛医院）；Stanford University Center for Artificial Intelligence in Medicine & Imaging（斯坦福大学医学与影像人工智能中心）；University of Montreal（蒙特利尔大学）
Medical images are often accompanied by metadata describing the image (vendor, acquisition parameters) and the patient (disease type or severity, demographics, genomics). This metadata is usually disregarded by image segmentation methods. In this work, we adapt a linear conditioning method called FiLM (Feature-wise Linear Modulation) for image segmentation tasks. This FiLM adaptation enables integrating metadata into segmentation models for better performance. We observed an average Dice score increase of 5.1% on spinal cord tumor segmentation when incorporating the tumor type with FiLM. The m…
Prior work on diagnosing Alzheimer's disease from magnetic resonance images of the brain established that convolutional neural networks (CNNs) can leverage the high-dimensional image information for classifying patients. However, little research focused on how these models can utilize the usually low-dimensional tabular information, such as patient demographics or laboratory measurements. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that dynamically rescales and shifts the feature maps of a convolutional layer, conditional on a patient's tabul…
CREATIS（CREATIS实验室）；Université Lyon1（里昂第一大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；INSA-Lyon（里昂国立应用科学学院）；Hospices Civils de Lyon（里昂民用医院）；CarMeN（CarMeN实验室）；INRA（法国国家农业研究院）；LARIS（LARIS实验室）；Universite d’Angers（昂热大学）
Clinical outcome prediction plays an important role in stroke patient management. From a machine learning point-of-view, one of the main challenges is dealing with heterogeneous data at patient admission, i.e. the image data which are multidimensional and the clinical data which are scalars. In this paper, a multimodal convolutional neural network - long short-term memory (CNN-LSTM) based ensemble model is proposed. For each MR image module, a dedicated network provides preliminary prediction of the clinical outcome using the modified Rankin scale (mRS). The final mRS score is obtained by merg…
Unidad Mixta de Imagen Biomédica FISABIO-CIPF（FISABIO-CIPF生物医学成像联合单元）；Universidad de Alicante（阿利坎特大学）；Medbravo；Hospital San Juan de Alicante（阿利坎特圣胡安医院）；Universidad Miguel Hernández（米格尔·埃尔南德斯大学）；Sierra Research SL（Sierra Research有限公司）；Bioinformatics & Biostatistics Unit, Principe Felipe Research Center（费利佩王子研究中心生物信息学与生物统计学单元）
This paper describes BIMCV COVID-19+, a large dataset from the Valencian Region Medical ImageBank (BIMCV) containing chest X-ray images CXR (CR, DX) and computed tomography (CT) imaging of COVID-19+ patients along with their radiological findings and locations, pathologies, radiological reports (in Spanish), DICOM metadata, Polymerase chain reaction (PCR), Immunoglobulin G (IgG) and Immunoglobulin M (IgM) diagnostic antibody tests. The findings have been mapped onto standard Unified Medical Language System (UMLS) terminology and cover a wide spectrum of thoracic entities, unlike the considerab…
kin cancer is considered one of the most common type of cancer in several countries. Due to the difficulty and subjectivity in the clinical diagnosis of skin lesions, Computer-Aided Diagnosis systems are being developed for assist experts to perform more reliable diagnosis. The clinical analysis and diagnosis of skin lesions relies not only on the visual information but also on the context information provided by the patient. This work addresses the problem of pigmented skin lesions detection from smartphones captured images. In addition to the features extracted from images, patient context i…
Assessing microsatellite stability status of a patient's colorectal cancer is crucial in personalizing treatment regime. Recently, convolutional-neural-networks (CNN) combined with transfer-learning approaches were proposed to circumvent traditional laboratory testing for determining microsatellite status from hematoxylin and eosin stained biopsy whole slide images (WSI). However, the high resolution of WSI practically prevent direct classification of the entire WSI. Current approaches bypass the WSI high resolution by first classifying small patches extracted from the WSI, and then aggregatin…
Skin cancer is one of the most common types of cancer around the world. For this reason, over the past years, different approaches have been proposed to assist detect it. Nonetheless, most of them are based only on dermoscopy images and do not take into account the patient clinical information. In this work, first, we present a new dataset that contains clinical images, acquired from smartphones, and patient clinical information of the skin lesions. Next, we introduce a straightforward approach to combine the clinical data and the images using different well-known deep learning models. These m…
Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
During the first wave of COVID-19, hospitals were overwhelmed with the high number of admitted patients. An accurate prediction of the most likely individual disease progression can improve the planning of limited resources and finding the optimal treatment for patients. However, when dealing with a newly emerging disease such as COVID-19, the impact of patient- and disease-specific factors (e.g. body weight or known co-morbidities) on the immediate course of disease is by and large unknown. In the case of COVID-19, the need for intensive care unit (ICU) admission of pneumonia patients is ofte…
KTH Royal Institute of Technology（瑞典皇家理工学院）；Science for Life Laboratory（生命科学实验室）；University of Tübingen（蒂宾根大学）
Privacy protection of medical image data is challenging. Even if metadata is removed, brain scans are vulnerable to attacks that match renderings of the face to facial image databases. Solutions have been developed to de-identify diagnostic scans by obfuscating or removing parts of the face. However, these solutions either fail to reliably hide the patient's identity or are so aggressive that they impair further analyses. We propose a new class of de-identification techniques that, instead of removing facial features, remodels them. Our solution relies on a conditional multi-scale GAN architec…
The use of ImageNet pre-trained networks is becoming widespread in the medical imaging community. It enables training on small datasets, commonly available in medical imaging tasks. The recent emergence of a large Chest X-ray dataset opened the possibility for learning features that are specific to the X-ray analysis task. In this work, we demonstrate that the features learned allow for better classification results for the problem of Tuberculosis detection and enable generalization to an unseen dataset. To accomplish the task of feature learning, we train a DenseNet-121 CNN on 112K images fro…
Deakin University（迪肯大学）；Islamic Azad University（伊斯兰阿扎德大学）；Isfahan University of Technology（伊斯法罕理工大学）；University of Birjand（比尔詹德大学）；Ferdowsi University of Mashhad（马什哈德菲尔多西大学）；K. N. Toosi University of Technology（K. N. 图西理工大学）；Universidad de Granada（格拉纳达大学）；University of Cambridge（剑桥大学）；Dibrugarh University（迪布鲁格尔大学）；Iran University of Medical Sciences（伊朗医科大学）；University of Bojnord（博杰纳德大学）；The George Institute for Global Health（乔治全球健康研究所）；University of Sydney（悉尼大学）
COVID-19 has caused many deaths worldwide. The automation of the diagnosis of this virus is highly desired. Convolutional neural networks (CNNs) have shown outstanding classification performance on image datasets. To date, it appears that COVID computer-aided diagnosis systems based on CNNs and clinical information have not yet been analysed or explored. We propose a novel method, named the CNN-AE, to predict the survival chance of COVID-19 patients using a CNN trained with clinical information. Notably, the required resources to prepare CT images are expensive and limited compared to those re…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）；Tata Medical Center（塔塔医学中心）；Arden Cancer Centre（雅顿癌症中心）；University Hospitals Coventry and Warwickshire（考文垂与沃里克郡大学医院）
Prediction of survivability in a patient for tumor progression is useful to estimate the effectiveness of a treatment protocol. In our work, we present a model to take into account the heterogeneous nature of a tumor to predict survival. The tumor heterogeneity is measured in terms of its mass by combining information regarding the radiodensity obtained in images with the gross tumor volume (GTV). We propose a novel feature called Tumor Mass within a GTV (TMG), that improves the prediction of survivability, compared to existing models which use GTV. Weekly variation in TMG of a patient is comp…
University of Virginia（弗吉尼亚大学）；Columbia University（哥伦比亚大学）
Eosinophilic Esophagitis (EoE) is an inflammatory esophageal disease which is increasing in prevalence. The diagnostic gold-standard involves manual review of a patient's biopsy tissue sample by a clinical pathologist for the presence of 15 or greater eosinophils within a single high-power field (400x magnification). Diagnosing EoE can be a cumbersome process with added difficulty for assessing the severity and progression of disease. We propose an automated approach for quantifying eosinophils using deep image segmentation. A U-Net model and post-processing system are applied to generate eosi…
University of Michigan（密歇根大学）；University of Michigan Medical School（密歇根大学医学院）
Objective: When patients develop acute respiratory failure, accurately identifying the underlying etiology is essential for determining the best treatment. However, differentiating between common medical diagnoses can be challenging in clinical practice. Machine learning models could improve medical diagnosis by aiding in the diagnostic evaluation of these patients. Materials and Methods: Machine learning models were trained to predict the common causes of acute respiratory failure (pneumonia, heart failure, and/or COPD). Models were trained using chest radiographs and clinical data from the e…
Medical head CT-scan imaging has been successfully combined with deep learning for medical diagnostics of head diseases and lesions[1]. State of the art classification models and algorithms for this task usually are based on 3d convolution layers for volumetric data on a supervised learning setting (1 input volume, 1 prediction per patient) or 2d convolution layers in a supervised setting (1 input image, 1 prediction per image). However a very common scenario in developing countries is to have the volume metadata lost due multiple reasons for example formatting conversion in images (for exampl…
Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification
Technical University of Munich（慕尼黑工业大学）；Munich Institute of Biomedical Engineering（慕尼黑生物医学工程研究所）；State Grid Corporation of China（国家电网有限公司）；Henan Economic Research Institute（河南经济研究院）；Imperial College London（帝国理工学院）；National Heart and Lung Institute（国家心肺研究所）；University of Zurich（苏黎世大学）；Department of Quantitative Biomedicine（定量生物医学系）；LMU Munich（慕尼黑大学）；University Hospital（大学医院）
Despite its high prevalence, anemia is often undetected due to the invasiveness and cost of screening and diagnostic tests. Though some non-invasive approaches have been developed, they are less accurate than invasive methods, resulting in an unmet need for more accurate non-invasive methods. Here, we show that deep learning-based algorithms can detect anemia and quantify several related blood measurements using retinal fundus images both in isolation and in combination with basic metadata such as patient demographics. On a validation dataset of 11,388 patients from the UK Biobank, our algorit…
GraphDerm: Fusing Imaging, Physical Scale, and Metadata in a Population-Graph Classifier for Dermoscopic Lesions
School of Computer Science, Institute for Research in Fundamental Sciences (IPM)（计算机科学学院，基础科学研究所（IPM））；Department of Physics, Shahid Beheshti University（物理系，肖德·贝赫谢什提大学）；Faculty of Science, University of Tehran（科学学院，德黑兰大学）；Fatemeh Zahra Hospital, Alborz University of Medical Sciences（法特梅·扎拉医院，阿尔bor兹医学科学大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系，成均馆大学）；Department of Computer Science and Engineering, Shahid Beheshti University（计算机科学与工程系，肖德·贝赫谢什提大学）；Autoimmune Bullous Diseases Research Center, Razi Hospital, Tehran University of Medical Sciences（自身免疫性水疱病研究中心，拉齐医院，德黑兰医学科学大学）
Birla Institute of Technology and Science Pilani（皮拉尼贝拉技术科学学院）
The objective of the study is to evaluate the efficiency of a multi layer neural network models built by combining Recurrent Neural Network(RNN) and Convolutional Neural Network(CNN) for solving the problem of classifying different types of White Blood Cells. This can have applications in the pharmaceutical and healthcare industry for automating the analysis of blood tests and other processes requiring identifying the nature of blood cells in a given image sample. It can also be used in the diagnosis of various blood-related diseases in patients.
Radiology Report Generation Using Transformers Conditioned with Non-imaging Data
Centre for Computational Imaging and Simulation Technologies in Biomedicine（生物医学计算成像与模拟技术中心）；Leeds Institute for Cardiovascular and Metabolic Medicine（利兹心血管与代谢医学研究所）；School of Medicine（医学院）；School of Computing, University of Leeds（利兹大学计算机学院）
本文提出一种多模态 Transformer 网络，整合胸部 X 射线图像与患者人口统计信息生成放射报告，实验证明加入非影像数据有效提升了生成质量。
LINE Corp（LINE公司）；NVIDIA（英伟达）；ReadSense Ltd（读感有限公司）
We present our winning solution to the SIIM-ISIC Melanoma Classification Challenge. It is an ensemble of convolutions neural network (CNN) models with different backbones and input sizes, most of which are image-only models while a few of them used image-level and patient-level metadata. The keys to our winning are: (1) stable validation scheme (2) good choice of model target (3) carefully tuned pipeline and (4) ensembling with very diverse models. The winning submission scored 0.9600 AUC on cross validation and 0.9490 AUC on private leaderboard.
A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data
University of Kentucky（肯塔基大学）；Department of Computer Science, University of Kentucky（肯塔基大学计算机科学系）；Department of Neurological Surgery, University of Kentucky（肯塔基大学神经外科学系）；University of Kentucky College of Medicine（肯塔基大学医学院）；Department of Neurology, University of Kentucky（肯塔基大学神经学系）
