Convolutional Neural Networks (CNNs) have shown to be powerful medical image segmentation models. In this study, we address some of the main unresolved issues regarding these models. Specifically, training of these models on small medical image datasets is still challenging, with many studies promoting techniques such as transfer learning. Moreover, these models are infamous for producing over-confident predictions and for failing silently when presented with out-of-distribution (OOD) data at test time. In this paper, we advocate for multi-task learning, i.e., training a single model on several different datasets, spanning several different organs of interest and different imaging modalities. We show that not only a single CNN learns to automatically recognize the context and accurately segment the organ of interest in each context, but also that such a joint model often has more accurate and better-calibrated predictions than dedicated models trained separately on each dataset. Our experiments show that multi-task learning can outperform transfer learning in medical image segmentation tasks. For detecting OOD data, we propose a method based on spectral analysis of CNN feature maps. We show that different datasets, representing different imaging modalities and/or different organs of interest, have distinct spectral signatures, which can be used to identify whether or not a test image is similar to the images used to train a model. We show that this approach is far more accurate than OOD detection based on prediction uncertainty. The methods proposed in this paper contribute significantly to improving the accuracy and reliability of CNN-based medical image segmentation models.
Recent works have shown that deep neural networks can achieve super-human performance in a wide range of image classification tasks in the medical imaging domain. However, these works have primarily focused on classification accuracy, ignoring the important role of uncertainty quantification. Empirically, neural networks are often miscalibrated and overconfident in their predictions. This miscalibration could be problematic in any automatic decision-making system, but we focus on the medical field in which neural network miscalibration has the potential to lead to significant treatment errors.…
National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）；Stanford University（斯坦福大学）
In medical image classification tasks, it is common to find that the number of normal samples far exceeds the number of abnormal samples. In such class-imbalanced situations, reliable training of deep neural networks continues to be a major challenge. Under these circumstances, the predicted class probabilities may be biased toward the majority class. Calibration has been suggested to alleviate some of these effects. However, there is insufficient analysis explaining when and whether calibrating a model would be beneficial in improving performance. In this study, we perform a systematic analys…
University of Florida（佛罗里达大学）；Herbert Wertheim College of Engineering（赫伯特·韦特海姆工程学院）；McKnight Brain Institute（麦克奈特脑研究所）；College of Public Health and Health Professions（公共卫生与健康职业学院）；College of Medicine（医学院）；United States Air Force Research Laboratory（美国空军研究实验室）
Model calibration measures the agreement between the predicted probability estimates and the true correctness likelihood. Proper model calibration is vital for high-risk applications. Unfortunately, modern deep neural networks are poorly calibrated, compromising trustworthiness and reliability. Medical image segmentation particularly suffers from this due to the natural uncertainty of tissue boundaries. This is exasperated by their loss functions, which favor overconfidence in the majority classes. We address these challenges with DOMINO, a domain-aware model calibration method that leverages …
In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, confidence estimates computed with a naive implementation of dropout can be poorly calibrated, particularly when using convolutional networks. In this paper, through the lens of ensemble learning, we associate calibration error with the correlation between the models sampled with dro…
Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation
Johns Hopkins University, School of Medicine（约翰霍普金斯大学医学院）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Healthcare & Life Sciences（微软医疗健康与生命科学）
Towards unraveling calibration biases in medical image analysis
Hospital Italiano de Buenos Aires（布宜诺斯艾利斯意大利医院）；Universidad Tecnológica Nacional（国立技术大学）；Research Institute for Signals, Systems and Computational Intelligence sinc(i) (FICH-UNL/CONICET)（信号、系统与计算智能研究所sinc(i)（FICH-UNL/阿根廷国家科学与技术研究理事会））
EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentation
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Applied Sciences Western Switzerland (HES-SO)（瑞士西部应用科学大学（HES-SO））
Maximum Entropy on Erroneous Predictions (MEEP): Improving model calibration for medical image segmentation
Tryolabs（Tryolabs公司）；Research institute for signals, systems and computational intelligence, sinc(i), FICH-UNL / CONICET（信号、系统与计算智能研究所（sinc(i)，国立拉普拉塔大学计算机科学学院/阿根廷国家科学与技术研究理事会））；LIVIA, ETS Montreal（蒙特利尔高等技术学院LIVIA实验室）；International Laboratory on Learning Systems (ILLS)（国际学习系统实验室（ILLS））
Deep neural networks offer numerous potential applications across geoscience, for example, one could argue that they are the state-of-the-art method for predicting faults in seismic datasets. In quantitative reservoir characterization workflows, it is common to incorporate the uncertainty of predictions thus such subsurface models should provide calibrated probabilities and the associated uncertainties in their predictions. It has been shown that popular Deep Learning-based models are often miscalibrated, and due to their deterministic nature, provide no means to interpret the uncertainty of t…
National Institute of Technology Karnataka（卡纳塔克邦国家技术研究所）
Computer-aided medical image analysis plays a significant role in assisting medical practitioners for expert clinical diagnosis and deciding the optimal treatment plan. At present, convolutional neural networks (CNN) are the preferred choice for medical image analysis. In addition, with the rapid advancements in three-dimensional (3D) imaging systems and the availability of excellent hardware and software support to process large volumes of data, 3D deep learning methods are gaining popularity in medical image analysis. Here, we present an extensive review of the recently evolved 3D deep learn…
Deep learning models are now used in many different industries, while in certain domains safety is not a critical issue in the medical field it is a huge concern. Not only, we want the models to generalize well but we also want to know the models confidence respect to its decision and which features matter the most. Our team aims to develop a full pipeline in which not only displays the uncertainty of the models decision but also, the saliency map to show which sets of pixels of the input image contribute most to the predictions.
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Sabaragamuwa University of Sri Lanka（斯里兰卡萨巴拉加穆瓦大学）；Digital Platform Development, SLT PLC（SLT公共有限公司数字平台开发部）
The shapes and morphology of the organs and tissues are important prior knowledge in medical imaging recognition and segmentation. The morphological operation is a well-known method for morphological feature extraction. As the morphological operation is performed well in hand-crafted image segmentation techniques, it is also promising to design an approach to approximate morphological operation in the convolutional networks. However, using the traditional convolutional neural network as a black-box is usually hard to specify the morphological operation action. Here, we introduced a 3D morpholo…
The University of British Columbia（不列颠哥伦比亚大学）；Harvard Medical School（哈佛医学院）
Fully convolutional neural networks (FCNs), and in particular U-Nets, have achieved state-of-the-art results in semantic segmentation for numerous medical imaging applications. Moreover, batch normalization and Dice loss have been used successfully to stabilize and accelerate training. However, these networks are poorly calibrated i.e. they tend to produce overconfident predictions both in correct and erroneous classifications, making them unreliable and hard to interpret. In this paper, we study predictive uncertainty estimation in FCNs for medical image segmentation. We make the following co…
Convolutional neural network-based medical image classifiers have been shown to be especially susceptible to adversarial examples. Such instabilities are likely to be unacceptable in the future of automated diagnoses. Though statistical adversarial example detection methods have proven to be effective defense mechanisms, additional research is necessary that investigates the fundamental vulnerabilities of deep-learning-based systems and how best to build models that jointly maximize traditional and robust accuracy. This paper presents the inclusion of attention mechanisms in CNN-based medical …
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；Siemens Healthineers（西门子医疗）
Although deep learning prediction models have been successful in the discrimination of different classes, they can often suffer from poor calibration across challenging domains including healthcare. Moreover, the long-tail distribution poses great challenges in deep learning classification problems including clinical disease prediction. There are approaches proposed recently to calibrate deep prediction in computer vision, but there are no studies found to demonstrate how the representative models work in different challenging contexts. In this paper, we bridge the confidence calibration from …
East China Normal University（华东师范大学）；Huashan Hospital, Fudan University（复旦大学附属华山医院）
Medical image segmentation is one of the fundamental problems for artificial intelligence-based clinical decision systems. Current automatic medical image segmentation methods are often failed to meet clinical requirements. As such, a series of interactive segmentation algorithms are proposed to utilize expert correction information. However, existing methods suffer from some segmentation refining failure problems after long-term interactions and some cost problems from expert annotation, which hinder clinical applications. This paper proposes an interactive segmentation framework, called inte…
Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification
Centre for Intelligent Machines, McGill University, Montreal, Canada（智能机器中心，麦吉尔大学，加拿大）；Mila -- Quebec AI Institute, Montreal, Canada（魁北克人工智能研究所）；Stanford University, Stanford, USA（斯坦福大学）；University of Copenhagen, Copenhagen, Denmark（哥本哈根大学）
In this paper, we propose a novel medical image segmentation using iterative deep learning framework. We have combined an iterative learning approach and an encoder-decoder network to improve segmentation results, which enables to precisely localize the regions of interest (ROIs) including complex shapes or detailed textures of medical images in an iterative manner. The proposed iterative deep convolutional encoder-decoder network consists of two main paths: convolutional encoder path and convolutional decoder path with iterative learning. Experimental results show that the proposed iterative …
University of Technology Sydney（悉尼科技大学）；School of Electrical and Data Engineering（电气与数据工程学院）
The Jaccard index, also known as Intersection-over-Union (IoU score), is one of the most critical evaluation metrics in medical image segmentation. However, directly optimizing the mean IoU (mIoU) score over multiple objective classes is an open problem. Although some algorithms have been proposed to optimize its surrogates, there is no guarantee provided for their generalization ability. In this paper, we present a novel data-distribution-aware margin calibration method for a better generalization of the mIoU over the whole data-distribution, underpinned by a rigid lower bound. This scheme en…
Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）
Transfer learning is widely used for training machine learning models. Here, we study the role of transfer learning for training fully convolutional networks (FCNs) for medical image segmentation. Our experiments show that although transfer learning reduces the training time on the target task, the improvement in segmentation accuracy is highly task/data-dependent. Larger improvements in accuracy are observed when the segmentation task is more challenging and the target training data is smaller. We observe that convolutional filters of an FCN change little during training for medical image seg…
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Zhihui Medical Technology (Shanghai) Co., Ltd.（智汇医疗科技（上海）有限公司）
Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Department of Radiology and Radiological Science（放射学与放射科学系）；Department of Electrical and Computer Engineering（电气与计算机工程系）
For the majority of the learning-based segmentation methods, a large quantity of high-quality training data is required. In this paper, we present a novel learning-based segmentation model that could be trained semi- or un- supervised. Specifically, in the unsupervised setting, we parameterize the Active contour without edges (ACWE) framework via a convolutional neural network (ConvNet), and optimize the parameters of the ConvNet using a self-supervised method. In another setting (semi-supervised), the auxiliary segmentation ground truth is used during training. We show that the method provide…
The Chinese University of Hong Kong（香港中文大学）；The First Hospital of Jilin University（吉林大学第一医院）；The Hong Kong Polytechnic University（香港理工大学）
Class distribution plays an important role in learning deep classifiers. When the proportion of each class in the test set differs from the training set, the performance of classification nets usually degrades. Such a label distribution shift problem is common in medical diagnosis since the prevalence of disease vary over location and time. In this paper, we propose the first method to tackle label shift for medical image classification, which effectively adapt the model learned from a single training label distribution to arbitrary unknown test label distribution. Our approach innovates distr…
We propose a novel technique to incorporate attention within convolutional neural networks using feature maps generated by a separate convolutional autoencoder. Our attention architecture is well suited for incorporation with deep convolutional networks. We evaluate our model on benchmark segmentation datasets in skin cancer segmentation and lung lesion segmentation. Results show highly competitive performance when compared with U-Net and it's residual variant.
Image-based characterization and disease understanding involve integrative analysis of morphological, spatial, and topological information across biological scales. The development of graph convolutional networks (GCNs) has created the opportunity to address this information complexity via graph-driven architectures, since GCNs can perform feature aggregation, interaction, and reasoning with remarkable flexibility and efficiency. These GCNs capabilities have spawned a new wave of research in medical imaging analysis with the overarching goal of improving quantitative disease understanding, mon…
Image denoising is an important pre-processing step in medical image analysis. Different algorithms have been proposed in past three decades with varying denoising performances. More recently, having outperformed all conventional methods, deep learning based models have shown a great promise. These methods are however limited for requirement of large training sample size and high computational costs. In this paper we show that using small sample size, denoising autoencoders constructed using convolutional layers can be used for efficient denoising of medical images. Heterogeneous images can be…
In this article, we look into some essential aspects of convolutional neural networks (CNNs) with the focus on medical image segmentation. First, we discuss the CNN architecture, thereby highlighting the spatial origin of the data, voxel-wise classification and the receptive field. Second, we discuss the sampling of input-output pairs, thereby highlighting the interaction between voxel-wise classification, patch size and the receptive field. Finally, we give a historical overview of crucial changes to CNN architectures for classification and segmentation, giving insights in the relation betwee…
University of Augsburg（奥格斯堡大学）；University Hospital Augsburg（奥格斯堡大学医院）
Neural networks represent a field of research that can quickly achieve very good results in the field of medical image segmentation using a GPU. A possible way to achieve good results without GPUs are random forests. For this purpose, two random forest approaches were compared with a state-of-the-art deep convolutional neural network. To make the comparison the PhC-C2DH-U373 and the retinal imaging datasets were used. The evaluation showed that the deep convolutional neutral network achieved the best results. However, one of the random forest approaches also achieved a similar high performance…
Massachusetts General Hospital（麻省总医院）；Zhejiang University（浙江大学）；University of Michigan（密歇根大学）；Beijing Institute of Technology（北京理工大学）
Tissue characterization has long been an important component of Computer Aided Diagnosis (CAD) systems for automatic lesion detection and further clinical planning. Motivated by the superior performance of deep learning methods on various computer vision problems, there has been increasing work applying deep learning to medical image analysis. However, the development of a robust and reliable deep learning model for computer-aided diagnosis is still highly challenging due to the combination of the high heterogeneity in the medical images and the relative lack of training samples. Specifically,…
Capsule-ConvKAN: A Hybrid Neural Approach to Medical Image Classification
Department of Cybernetics and Artificial Intelligence, Technical University of Košice（控制论与人工智能系，科希斯技术大学）；Institute of Information Science, University of Miskolc（信息科学研究所，米什科尔特大学）；Department of Computer Science and Engineering, University of Connecticut（计算机科学与工程系，康涅狄格大学）
Calibration-Aware Prompt Learning for Medical Vision-Language Models
Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（计算机视觉系，Mohamed bin Zayed人工智能大学）；Department of Computer Science and Engineering, Michigan State University (MSU)（计算机科学与工程系，密歇根州立大学）
The Hong Kong Polytechnic University（香港理工大学）；The University of Tokyo（东京大学）
Spatial attention has been introduced to convolutional neural networks (CNNs) for improving both their performance and interpretability in visual tasks including image classification. The essence of the spatial attention is to learn a weight map which represents the relative importance of activations within the same layer or channel. All existing attention mechanisms are local attentions in the sense that weight maps are image-specific. However, in the medical field, there are cases that all the images should share the same weight map because the set of images record the same kind of symptom r…
We propose a novel transformer model, capable of segmenting medical images of varying modalities. Challenges posed by the fine grained nature of medical image analysis mean that the adaptation of the transformer for their analysis is still at nascent stages. The overwhelming success of the UNet lay in its ability to appreciate the fine-grained nature of the segmentation task, an ability which existing transformer based models do not currently posses. To address this shortcoming, we propose The Fully Convolutional Transformer (FCT), which builds on the proven ability of Convolutional Neural Net…
Segmentation is a critical step in medical image analysis. Fully Convolutional Networks (FCNs) have emerged as powerful segmentation models achieving state-of-the-art results in various medical image datasets. Network architectures are usually designed manually for a specific segmentation task so applying them to other medical datasets requires extensive experience and time. Moreover, the segmentation requires handling large volumetric data that results in big and complex architectures. Recently, methods that automatically design neural networks for medical image segmentation have been present…
University of Sydney（悉尼大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）；Shanghai Jiao Tong University（上海交通大学）
Medical image analysis using supervised deep learning methods remains problematic because of the reliance of deep learning methods on large amounts of labelled training data. Although medical imaging data repositories continue to expand there has not been a commensurate increase in the amount of annotated data. Hence, we propose a new unsupervised feature learning method that learns feature representations to then differentiate dissimilar medical images using an ensemble of different convolutional neural networks (CNNs) and K-means clustering. It jointly learns feature representations and clus…
A Distributed Hybrid Quantum Convolutional Neural Network for Medical Image Classification
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；Joint International Research Laboratory of Intelligent Perception and Computation（智能感知与计算联合国际研究实验室）；Collaborative Innovation Center of Quantum Information of Shaanxi Province（陕西省量子信息协同创新中心）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）
University of Bucharest（布加勒斯特大学）；“Carol Davila” University of Medicine and Pharmacy（“卡罗尔·达维拉”医药大学）；Colţea Hospital（科尔察医院）；University Politehnica of Bucharest（布加勒斯特理工大学）；MBZ University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Linköping University（林雪平大学）
Super-resolving medical images can help physicians in providing more accurate diagnostics. In many situations, computed tomography (CT) or magnetic resonance imaging (MRI) techniques capture several scans (modes) during a single investigation, which can jointly be used (in a multimodal fashion) to further boost the quality of super-resolution results. To this end, we propose a novel multimodal multi-head convolutional attention module to super-resolve CT and MRI scans. Our attention module uses the convolution operation to perform joint spatial-channel attention on multiple concatenated input …
Case Western Reserve University（凯斯西储大学）；IBM Almaden Research Center（IBM阿尔马登研究中心）
We explore a solution for learning disease signatures from weakly, yet easily obtainable, annotated volumetric medical imaging data by analyzing 3D volumes as a sequence of 2D images. We demonstrate the performance of our solution in the detection of emphysema in lung cancer screening low-dose CT images. Our approach utilizes convolutional long short-term memory (LSTM) to "scan" sequentially through an imaging volume for the presence of disease in a portion of scanned region. This framework allowed effective learning given only volumetric images and binary disease labels, thus enabling trainin…
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious, laborious, and time consuming, but it also demands costly, specialty-oriented skills, which are not easily accessible. To dramatically reduce annotation cost, this paper presents a novel method to naturally integrate active learning and transfer learning (fine-tuning) into a si…
University of Engineering and Technology Taxila（塔克西拉工程技术大学）；University of Surrey（萨里大学）；King Abdul Aziz University（阿卜杜勒·阿齐兹国王大学）；King Saud University（沙特国王大学）
The science of solving clinical problems by analyzing images generated in clinical practice is known as medical image analysis. The aim is to extract information in an effective and efficient manner for improved clinical diagnosis. The recent advances in the field of biomedical engineering has made medical image analysis one of the top research and development area. One of the reason for this advancement is the application of machine learning techniques for the analysis of medical images. Deep learning is successfully used as a tool for machine learning, where a neural network is capable of au…
CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation
Shaanxi University of Science and Technology（陕西科技大学）；Xi’an Jiaotong University（西安交通大学）；Northwestern Polytechnical University（西北工业大学）；Brunel University London（伦敦布鲁内尔大学）
Residual network (ResNet) and densely connected network (DenseNet) have significantly improved the training efficiency and performance of deep convolutional neural networks (DCNNs) mainly for object classification tasks. In this paper, we propose an efficient network architecture by considering advantages of both networks. The proposed method is integrated into an encoder-decoder DCNN model for medical image segmentation. Our method adds additional skip connections compared to ResNet but uses significantly fewer model parameters than DenseNet. We evaluate the proposed method on a public datase…
