Self-Supervised Learning for 3D Medical Image Analysis using 3D SimCLR and Monte Carlo Dropout
Technical University of Berlin, Germany（柏林工业大学（德国））；Hasso Plattner Institute for Digital Engineering, University of Potsdam, Germany（波茨坦大学哈索普拉特纳数字工程研究所（德国））；Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai, NYC, USA（西奈山伊坎医学院哈索普拉特纳数字健康研究所（美国纽约市））
Yamen Ali, Aiham Taleb, Marina M. -C. Höhne, Christoph Lippert
Self-supervised learning methods can be used to learn meaningful representations from unlabeled data that can be transferred to supervised downstream tasks to reduce the need for labeled data. In this paper, we propose a 3D self-supervised method that is based on the contrastive (SimCLR) method. Additionally, we show that employing Bayesian neural networks (with Monte-Carlo Dropout) during the inference phase can further enhance the results on the downstream tasks. We showcase our models on two medical imaging segmentation tasks: i) Brain Tumor Segmentation from 3D MRI, ii) Pancreas Tumor Segmentation from 3D CT. Our experimental results demonstrate the benefits of our proposed methods in both downstream data-efficiency and performance.
Northeastern University（东北大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
There are limited works showing the efficacy of unsupervised Out-of-Distribution (OOD) methods on complex medical data. Here, we present preliminary findings of our unsupervised OOD detection algorithm, SimCLR-LOF, as well as a recent state of the art approach (SSD), applied on medical images. SimCLR-LOF learns semantically meaningful features using SimCLR and uses LOF for scoring if a test sample is OOD. We evaluated on the multi-source International Skin Imaging Collaboration (ISIC) 2019 dataset, and show results that are competitive with SSD as well as with recent supervised approaches appl…
Johns Hopkins University（约翰斯·霍普金斯大学）；Sigma Technologies（12 Sigma科技公司）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
Medical images are increasingly used as input to deep neural networks to produce quantitative values that aid researchers and clinicians. However, standard deep neural networks do not provide a reliable measure of uncertainty in those quantitative values. Recent work has shown that using dropout during training and testing can provide estimates of uncertainty. In this work, we investigate using dropout to estimate epistemic and aleatoric uncertainty in a CT-to-MR image translation task. We show that both types of uncertainty are captured, as defined, providing confidence in the output uncertai…
Regularization in convolutional neural networks (CNNs) is usually addressed with dropout layers. However, dropout is sometimes detrimental in the convolutional part of a CNN as it simply sets to zero a percentage of pixels in the feature maps, adding unrepresentative examples during training. Here, we propose a CNN layer that performs regularization by applying random rotations of reflections to a small percentage of feature maps after every convolutional layer. We prove how this concept is beneficial for images with orientational symmetries, such as in medical images, as it provides a certain…
KTH Royal Institute of Technology（瑞典皇家理工学院）；Science for Life Laboratory（生命科学实验室）；Karolinska Institutet（卡罗林斯卡学院）；Karolinska University Hospital（卡罗林斯卡大学医院）；AstraZeneca（阿斯利康）
Vision transformers have demonstrated the potential to outperform CNNs in a variety of vision tasks. But the computational and memory requirements of these models prohibit their use in many applications, especially those that depend on high-resolution images, such as medical image classification. Efforts to train ViTs more efficiently are overly complicated, necessitating architectural changes or intricate training schemes. In this work, we show that standard ViT models can be efficiently trained at high resolution by randomly dropping input image patches. This simple approach, PatchDropout, r…
Massachusetts General Hospital（麻省总医院）；Polytechnique Montreal（蒙特利尔理工学院）；Mila, Quebec AI Institute（Mila魁北克人工智能研究所）；Massachusetts Institute of Technology（麻省理工学院）；MGH and BWH Center for Clinical Data Science（麻省总医院与布莱根妇女医院临床数据科学中心）；National Cancer Institute（美国国家癌症研究所）；Oregon Health and Science University（俄勒冈健康与科学大学）
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Among the main features of robustness is repeatability. Much attention is given to classification performance without assessing the model repeatability, leading to the development of models that turn out to be unusable in practice. In this work, we evaluate the repeatability of four model types on images from the same patient that were acquired during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on three medical image analysis tasks: cervica…
Although automated pathology classification using deep learning (DL) has proved to be predictively efficient, DL methods are found to be data and compute cost intensive. In this work, we aim to reduce DL training costs by pre-training a Resnet feature extractor using SimCLR contrastive loss for latent encoding of OCT images. We propose a novel active learning framework that identifies a minimal sub-sampled dataset containing the most uncertain OCT image samples using label propagation on the SimCLR latent encodings. The pre-trained Resnet model is then fine-tuned with the labelled minimal sub-…
Enhancing Global Sensitivity and Uncertainty Quantification in Medical Image Reconstruction with Monte Carlo Arbitrary-Masked Mamba
Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区生物医学工程学院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；Inselspital, University of Bern（伯尔尼大学因塞尔医院）；Technical University of Munich（慕尼黑工业大学）
Jadavpur University（贾达沃普尔大学）；University of Toronto（多伦多大学）
Privacy and annotation bottlenecks are two major issues that profoundly affect the practicality of machine learning-based medical image analysis. Although significant progress has been made in these areas, these issues are not yet fully resolved. In this paper, we seek to tackle these concerns head-on and systematically explore the applicability of non-contrastive self-supervised learning (SSL) algorithms under federated learning (FL) simulations for medical image analysis. We conduct thorough experimentation of recently proposed state-of-the-art non-contrastive frameworks under standard FL se…
Self-supervised Visualisation of Medical Image Datasets
Hertie Institute for AI in Brain Health（赫蒂脑健康人工智能研究院）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Heidelberg University（海德堡大学）；IWR, Heidelberg University（海德堡大学IWR研究所）
Deep neural networks have become the gold-standard approach for the automated segmentation of 3D medical images. Their full acceptance by clinicians remains however hampered by the lack of intelligible uncertainty assessment of the provided results. Most approaches to quantify their uncertainty, such as the popular Monte Carlo dropout, restrict to some measure of uncertainty in prediction at the voxel level. In addition not to be clearly related to genuine medical uncertainty, this is not clinically satisfying as most objects of interest (e.g. brain lesions) are made of groups of voxels whose …
Using Multi-Instance Learning to Identify Unique Polyps in Colon Capsule Endoscopy Images
Department of Automation and Process Engineering, Machine Learning Group（自动化与过程工程系，机器学习组）；UiT – The Arctic University of Norway（UiT – 北极大学）；Department of Physics and Technology, Machine Learning Group（物理与技术系，机器学习组）；University of Waikato（瓦卡托大学）；Odense University Hospital（奥登塞大学医院）；University of Southern Denmark（南丹麦大学）
Yale University（耶鲁大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Automated segmentation in medical image analysis is a challenging task that requires a large amount of manually labeled data. However, most existing learning-based approaches usually suffer from limited manually annotated medical data, which poses a major practical problem for accurate and robust medical image segmentation. In addition, most existing semi-supervised approaches are usually not robust compared with the supervised counterparts, and also lack explicit modeling of geometric structure and semantic information, both of which limit the segmentation accuracy. In this work, we present S…
University College London（伦敦大学学院）；King’s College London（伦敦国王学院）；University of Electronic Science and Technology of China（电子科技大学）；University Hospitals Leuven（鲁汶大学医院）；Institute for Women’s Health（妇女健康研究所）
Despite the state-of-the-art performance for medical image segmentation, deep convolutional neural networks (CNNs) have rarely provided uncertainty estimations regarding their segmentation outputs, e.g., model (epistemic) and image-based (aleatoric) uncertainties. In this work, we analyze these different types of uncertainties for CNN-based 2D and 3D medical image segmentation tasks. We additionally propose a test-time augmentation-based aleatoric uncertainty to analyze the effect of different transformations of the input image on the segmentation output. Test-time augmentation has been previo…
Deep neural networks are increasingly being used for the analysis of medical images. However, most works neglect the uncertainty in the model's prediction. We propose an uncertainty-aware deep kernel learning model which permits the estimation of the uncertainty in the prediction by a pipeline of a Convolutional Neural Network and a sparse Gaussian Process. Furthermore, we adapt different pre-training methods to investigate their impacts on the proposed model. We apply our approach to Bone Age Prediction and Lesion Localization. In most cases, the proposed model shows better performance compar…
Hamburg University of Technology（汉堡工业大学）；Leibniz Universität Hannover（莱布尼茨汉诺威大学）；Hannover Medical School（汉诺威医学院）；The Hospital for Sick Children（多伦多病童医院）；University of Toronto Mississauga（多伦多大学密西沙加校区）
The consideration of predictive uncertainty in medical imaging with deep learning is of utmost importance. We apply estimation of both aleatoric and epistemic uncertainty by variational Bayesian inference with Monte Carlo dropout to regression tasks and show that predictive uncertainty is systematically underestimated. We apply $ σ$ scaling with a single scalar value; a simple, yet effective calibration method for both types of uncertainty. The performance of our approach is evaluated on a variety of common medical regression data sets using different state-of-the-art convolutional network arc…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Wake Forest School of Medicine（维克森林医学院）
Recently, deep learning has achieved remarkable successes in medical image analysis. Although deep neural networks generate clinically important predictions, they have inherent uncertainty. Such uncertainty is a major barrier to report these predictions with confidence. In this paper, we propose a novel yet simple Bayesian inference approach called SoftDropConnect (SDC) to quantify the network uncertainty in medical imaging tasks with gliomas segmentation and metastases classification as initial examples. Our key idea is that during training and testing SDC modulates network parameters continu…
We explore different curriculum learning methods for training convolutional neural networks on the task of deformable pairwise 3D medical image registration. To the best of our knowledge, we are the first to attempt to improve performance by training medical image registration models using curriculum learning, starting from an easy training setup in the first training stages, and gradually increasing the complexity of the setup. On the one hand, we consider two existing curriculum learning approaches, namely curriculum dropout and curriculum by smoothing. On the other hand, we propose a novel …
No Modality Left Behind: Dynamic Model Generation for Incomplete Medical Data
Computational Imaging Research Lab（计算成像研究实验室）；Department for Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Medical University of Vienna（维也纳医学大学）；Comprehensive Center for Artificial Intelligence in Medicine（医学人工智能综合中心）；Christian Doppler Laboratory for Machine Learning Driven Precision Imaging（机器学习驱动精准成像的克里斯蒂安·多普勒实验室）；Department of Biomedical Imaging and Image-guided Therapy（生物医学成像与影像引导治疗部门）；Athinoula A. Martinos Center for Biomedical Imaging（阿提诺拉·A·马丁努斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Department of Radiology（放射科）；Division of General and Pediatric Radiology（普通和儿童放射科部门）
Towards Trustworthy Breast Tumor Segmentation in Ultrasound using Monte Carlo Dropout and Deep Ensembles for Epistemic Uncertainty Estimation
Department of Computer Engineering, Kwame Nkrumah University of Science and Technology（计算机工程系，加纳科拉布大学科学与技术大学）；Global Health and Infectious Disease Group, Kumasi Centre for Collaborative Research in Tropical Medicine（全球卫生与传染病组，科拉布热带医学协作研究中心）；Department of Radiology, National Hospital Abuja（放射科，阿布贾国家医院）；Computer Science Department, FAST National University of Computer and Emerging Sciences（计算机科学系，FAST国家计算机与新兴科学大学）；Medical Artificial Intelligence Lab, Lagos, Nigeria（医学人工智能实验室，尼日利亚拉各斯）；Department of Biomedical Engineering, McGill University（生物医学工程系，麦吉尔大学）；Perelman School of Medicine, University of Pennsylvania（佩尔曼医学学院，宾夕法尼亚大学）；Nnamdi Azikiwe University Teaching Hospital（纳曼迪·阿齐基韦大学教学医院）
Martinos Center for Biomedical Imaging（马蒂诺斯生物医学成像中心）；NeuroPoly, Polytechnique Montreal（蒙特利尔理工学院神经聚合物实验室）；Massachusetts Institute of Technology（麻省理工学院）；MGH & BWH Center for Clinical Data Science（麻省总医院与布莱根妇女医院临床数据科学中心）；University of Washington School of Public Health（华盛顿大学公共卫生学院）；National Cancer Institute（美国国家癌症研究所）；Oregon Health and Science University（俄勒冈健康与科学大学）
The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Repeatable models output predictions with low variation during independent tests carried out under similar conditions. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class class…
Uncertainty Estimation for Pretrained Medical Image Registration Models via Transformation Equivariance
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；The Hawkes Institute, University College London（霍克斯研究所，伦敦大学学院）；AI Laboratory, Massachusetts Institute of Technology（人工智能实验室，麻省理工学院）
Elekta（医科达）；KTH - Royal institute of Technology（KTH皇家理工学院）
Over the last few years machine learning has demonstrated groundbreaking results in many areas of medical image analysis, including segmentation. A key assumption, however, is that the train- and test distributions match. We study a realistic scenario where this assumption is clearly violated, namely segmentation with missing input modalities. We describe two neural network approaches that can handle a variable number of input modalities. The first is modality dropout: a simple but surprisingly effective modification of the training. The second is the unified representation network: a network …
Airdoc（鹰瞳科技）；Harbin Engineering University（哈尔滨工程大学）；IIAI（国际人工智能研究院）；Monash University（莫纳什大学）；The University of Sydney（悉尼大学）
Deep neural networks are known to be data-driven and label noise can have a marked impact on model performance. Recent studies have shown great robustness to classic image recognition even under a high noisy rate. In medical applications, learning from datasets with label noise is more challenging since medical imaging datasets tend to have asymmetric (class-dependent) noise and suffer from high observer variability. In this paper, we systematically discuss and define the two common types of label noise in medical images - disagreement label noise from inconsistency expert opinions and single-…
Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心图像科学研究所）；Department of Biomedical Engineering, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）
Despite the successes of deep learning techniques at detecting objects in medical images, false positive detections occur which may hinder an accurate diagnosis. We propose a technique to reduce false positive detections made by a neural network using an SVM classifier trained with features derived from the uncertainty map of the neural network prediction. We demonstrate the effectiveness of this method for the detection of liver lesions on a dataset of abdominal MR images. We find that the use of a dropout rate of 0.5 produces the least number of false positives in the neural network predicti…
Deep learning (DL) networks have recently been shown to outperform other segmentation methods on various public, medical-image challenge datasets [3,11,16], especially for large pathologies. However, in the context of diseases such as Multiple Sclerosis (MS), monitoring all the focal lesions visible on MRI sequences, even very small ones, is essential for disease staging, prognosis, and evaluating treatment efficacy. Moreover, producing deterministic outputs hinders DL adoption into clinical routines. Uncertainty estimates for the predictions would permit subsequent revision by clinicians. We …
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Atlas-based methods are the standard approaches for automatic targeting of the Anterior Nucleus of the Thalamus (ANT) for Deep Brain Stimulation (DBS), but these are known to lack robustness when anatomic differences between atlases and subjects are large. To improve the localization robustness, we propose a novel two-stage deep learning (DL) framework, where the first stage identifies and crops the thalamus regions from the whole brain MRI and the second stage performs per-voxel regression on the cropped volume to localize the targets at the finest resolution scale. To address the issue of da…
University of Toronto（多伦多大学）；University of British Columbia（不列颠哥伦比亚大学）；Sunnybrook Research Institute（森尼布鲁克研究所）
Unsupervised learning has been a long-standing goal of machine learning and is especially important for medical image analysis, where the learning can compensate for the scarcity of labeled datasets. A promising subclass of unsupervised learning is self-supervised learning, which aims to learn salient features using the raw input as the learning signal. In this paper, we use a contrastive self-supervised learning method called SimCLR that achieved state-of-the-art results on natural-scene images and apply this method to digital histopathology by collecting and pretraining on 57 histopathology …
We propose to learn a probabilistic motion model from a sequence of images. Besides spatio-temporal registration, our method offers to predict motion from a limited number of frames, useful for temporal super-resolution. The model is based on a probabilistic latent space and a novel temporal dropout training scheme. This enables simulation and interpolation of realistic motion patterns given only one or any subset of frames of a sequence. The encoded motion also allows to be transported from one subject to another without the need of inter-subject registration. An unsupervised generative defor…
Georgia Institute of Technology（佐治亚理工学院）；TReNDS Center（TReNDS中心）；Delft University of Technology（代尔夫特理工大学）
Self-supervised learning has enabled significant improvements on natural image benchmarks. However, there is less work in the medical imaging domain in this area. The optimal models have not yet been determined among the various options. Moreover, little work has evaluated the current applicability limits of novel self-supervised methods. In this paper, we evaluate a range of current contrastive self-supervised methods on out-of-distribution generalization in order to evaluate their applicability to medical imaging. We show that self-supervised models are not as robust as expected based on the…
National University of Defense Technology（国防科技大学）；Wuhan University（武汉大学）
Self-supervised pre-training has become the priory choice to establish reliable neural networks for automated recognition of massive biomedical microscopy images, which are routinely annotation-free, without semantics, and without guarantee of quality. Note that this paradigm is still at its infancy and limited by closely related open issues: 1) how to learn robust representations in an unsupervised manner from unlabelled biomedical microscopy images of low diversity in samples? and 2) how to obtain the most significant representations demanded by a high-quality segmentation? Aiming at these i…
University of Campinas（坎皮纳斯大学）；Utrecht University（乌得勒支大学）
Lacking supervised data is an issue while training deep neural networks (DNNs), mainly when considering medical and biological data where supervision is expensive. Recently, Embedded Pseudo-Labeling (EPL) addressed this problem by using a non-linear projection (t-SNE) from a feature space of the DNN to a 2D space, followed by semi-supervised label propagation using a connectivity-based method (OPFSemi). We argue that the performance of the final classifier depends on the data separation present in the latent space and visual separation present in the projection. We address this by first propos…
Self-supervised methods have shown tremendous success in the field of computer vision, including applications in remote sensing and medical imaging. Most popular contrastive-loss based methods like SimCLR, MoCo, MoCo-v2 use multiple views of the same image by applying contrived augmentations on the image to create positive pairs and contrast them with negative examples. Although these techniques work well, most of these techniques have been tuned on ImageNet (and similar computer vision datasets). While there have been some attempts to capture a richer set of deformations in the positive sampl…
Purpose: Considering several patients screened due to COVID-19 pandemic, computer-aided detection has strong potential in assisting clinical workflow efficiency and reducing the incidence of infections among radiologists and healthcare providers. Since many confirmed COVID-19 cases present radiological findings of pneumonia, radiologic examinations can be useful for fast detection. Therefore, chest radiography can be used to fast screen COVID-19 during the patient triage, thereby determining the priority of patient's care to help saturated medical facilities in a pandemic situation. Methods: I…
National Cheng Kung University（台湾成功大学）；Kun Shan University（昆山科技大学）；National Cheng Kung University Hospital（台湾成功大学医院）
In recent years, convolutional neural networks (CNNs) have been successfully implemented to various image recognition applications, such as medical image analysis, object detection, and image segmentation. Many studies and applications have been working on improving the performance of CNN algorithms and models. The strategies that aim to improve the performance of CNNs can be grouped into three major approaches: (1) deeper and wider network architecture, (2) automatic architecture search, and (3) convolutional attention block. Unlike approaches (1) and (2), the convolutional attention block ap…
Universitat Politècnica de Catalunya. BarcelonaTech（加泰罗尼亚理工大学·巴塞罗那理工）
Automation of brain tumor segmentation in 3D magnetic resonance images (MRIs) is key to assess the diagnostic and treatment of the disease. In recent years, convolutional neural networks (CNNs) have shown improved results in the task. However, high memory consumption is still a problem in 3D-CNNs. Moreover, most methods do not include uncertainty information, which is especially critical in medical diagnosis. This work studies 3D encoder-decoder architectures trained with patch-based techniques to reduce memory consumption and decrease the effect of unbalanced data. The different trained model…
Deep Learning-Based Pneumonia Detection from Chest X-ray Images: A CNN Approach with Performance Analysis and Clinical Implications
Amity Institute of Bio Technology, Amity University Kolkata（阿米蒂大学加尔各答分校阿米蒂生物技术学院）；Dept of Computer Science, Heritage Institute of Technology, Kolkata（加尔各答遗产技术学院计算机科学系）；School of Engineering and Technology, Amity University Kolkata（阿米蒂大学加尔各答分校工程与技术学院）
