Although deep neural networks have been a dominant method for many 2D vision tasks, it is still challenging to apply them to 3D tasks, such as medical image segmentation, due to the limited amount of annotated 3D data and limited computational resources. In this chapter, by rethinking the strategy to apply 3D Convolutional Neural Networks to segment medical images, we propose a novel 3D-based coarse-to-fine framework to efficiently tackle these challenges. The proposed 3D-based framework outperforms their 2D counterparts by a large margin since it can leverage the rich spatial information along all three axes. We further analyze the threat of adversarial attacks on the proposed framework and show how to defense against the attack. We conduct experiments on three datasets, the NIH pancreas dataset, the JHMI pancreas dataset and the JHMI pathological cyst dataset, where the first two and the last one contain healthy and pathological pancreases respectively, and achieve the current state-of-the-art in terms of Dice-Sorensen Coefficient (DSC) on all of them. Especially, on the NIH pancreas segmentation dataset, we outperform the previous best by an average of over $2\%$, and the worst case is improved by $7\%$ to reach almost $70\%$, which indicates the reliability of our framework in clinical applications.
We propose an attention mechanism for 3D medical image segmentation. The method, named segmentation-by-detection, is a cascade of a detection module followed by a segmentation module. The detection module enables a region of interest to come to attention and produces a set of object region candidates which are further used as an attention model. Rather than dealing with the entire volume, the segmentation module distills the information from the potential region. This scheme is an efficient solution for volumetric data as it reduces the influence of the surrounding noise which is especially im…
Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai University（上海大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
In this paper, we adopt 3D Convolutional Neural Networks to segment volumetric medical images. Although deep neural networks have been proven to be very effective on many 2D vision tasks, it is still challenging to apply them to 3D tasks due to the limited amount of annotated 3D data and limited computational resources. We propose a novel 3D-based coarse-to-fine framework to effectively and efficiently tackle these challenges. The proposed 3D-based framework outperforms the 2D counterpart to a large margin since it can leverage the rich spatial infor- mation along all three axes. We conduct ex…
Volumetric Medical Image Segmentation via Scribble Annotations and Shape Priors
Shanghai Jiao Tong University（上海交通大学）；Ruijin Hospital, Shanghai Jiao Tong University of Medicine（上海交通大学医学院附属瑞金医院）；Shanghai Jiao Tong University of Medicine（上海交通大学医学院）
The recent 3D medical ViTs (e.g., SwinUNETR) achieve the state-of-the-art performances on several 3D volumetric data benchmarks, including 3D medical image segmentation. Hierarchical transformers (e.g., Swin Transformers) reintroduced several ConvNet priors and further enhanced the practical viability of adapting volumetric segmentation in 3D medical datasets. The effectiveness of hybrid approaches is largely credited to the large receptive field for non-local self-attention and the large number of model parameters. In this work, we propose a lightweight volumetric ConvNet, termed 3D UX-Net, w…
We consider the problem of localizing and segmenting individual teeth inside 3D Cone-Beam Computed Tomography (CBCT) images. To handle large image sizes we approach this task with a coarse-to-fine framework, where the whole volume is first analyzed as a 33-class semantic segmentation (adults have up to 32 teeth) in coarse resolution, followed by binary semantic segmentation of the cropped region of interest in original resolution. To improve the performance of the challenging 33-class segmentation, we first train the Coarse step model on a large weakly labeled dataset, then fine-tune it on a s…
University of Pennsylvania（宾夕法尼亚大学）；Ryerson University（瑞尔森大学）
This paper addresses the challenge of 3D human pose estimation from a single color image. Despite the general success of the end-to-end learning paradigm, top performing approaches employ a two-step solution consisting of a Convolutional Network (ConvNet) for 2D joint localization and a subsequent optimization step to recover 3D pose. In this paper, we identify the representation of 3D pose as a critical issue with current ConvNet approaches and make two important contributions towards validating the value of end-to-end learning for this task. First, we propose a fine discretization of the 3D …
Recently, several Bayesian deep learning methods have been proposed for semi-supervised medical image segmentation. Although they have achieved promising results on medical benchmarks, some problems are still existing. Firstly, their overall architectures belong to the discriminative models, and hence, in the early stage of training, they only use labeled data for training, which might make them overfit to the labeled data. Secondly, in fact, they are only partially based on Bayesian deep learning, as their overall architectures are not designed under the Bayesian framework. However, unifying …
Sigma Technologies（12 Sigma科技公司）；Univ. of California, San Diego（加州大学圣地亚哥分校）
A volumetric attention(VA) module for 3D medical image segmentation and detection is proposed. VA attention is inspired by recent advances in video processing, enables 2.5D networks to leverage context information along the z direction, and allows the use of pretrained 2D detection models when training data is limited, as is often the case for medical applications. Its integration in the Mask R-CNN is shown to enable state-of-the-art performance on the Liver Tumor Segmentation (LiTS) Challenge, outperforming the previous challenge winner by 3.9 points and achieving top performance on the LiTS …
In the isointense stage, the accurate volumetric image segmentation is a challenging task due to the low contrast between tissues. In this paper, we propose a novel very deep network architecture based on a densely convolutional network for volumetric brain segmentation. The proposed network architecture provides a dense connection between layers that aims to improve the information flow in the network. By concatenating features map of fine and coarse dense blocks, it allows capturing multi-scale contextual information. Experimental results demonstrate significant advantages of the proposed me…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；NVIDIA（英伟达）
Textures and edges contribute different information to image recognition. Edges and boundaries encode shape information, while textures manifest the appearance of regions. Despite the success of Convolutional Neural Networks (CNNs) in computer vision and medical image analysis applications, predominantly only texture abstractions are learned, which often leads to imprecise boundary delineations. In medical imaging, expert manual segmentation often relies on organ boundaries; for example, to manually segment a liver, a medical practitioner usually identifies edges first and subsequently fills i…
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Universidad de los Andes（洛斯安第斯大学）
The reliability of Deep Learning systems depends on their accuracy but also on their robustness against adversarial perturbations to the input data. Several attacks and defenses have been proposed to improve the performance of Deep Neural Networks under the presence of adversarial noise in the natural image domain. However, robustness in computer-aided diagnosis for volumetric data has only been explored for specific tasks and with limited attacks. We propose a new framework to assess the robustness of general medical image segmentation systems. Our contributions are two-fold: (i) we propose a…
BiSegMamba: Efficient Bidirectional Tri-Oriented Mamba for 3D Medical Image Segmentation
School of Computer Science and Engineering, Beihang University（北航计算机科学与工程学院）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北航虚拟现实技术与系统国家重点实验室）
Deep Convolutional Neural Networks (CNNs) i.e. Residual Networks (ResNets) have been used successfully for many computer vision tasks, but are difficult to scale to 3D volumetric medical data. Memory is increasingly often the bottleneck when training 3D Convolutional Neural Networks (CNNs). Recently, invertible neural networks have been applied to significantly reduce activation memory footprint when training neural networks with backpropagation thanks to the invertible functions that allow retrieving its input from its output without storing intermediate activations in memory to perform the b…
RefineSeg: Dual Coarse-to-Fine Learning for Medical Image Segmentation
School of Engineering, College of Engineering and Physical Sciences（工程学院，工程与物理科学学院）；University of Birmingham（伯明翰大学）；Oxford Center for Clinical Magnetic Resonance Research (OCMR)（牛津临床磁共振研究所以（OCMR））；University of Oxford（牛津大学）；John Radcliffe Hospital（约翰·拉德克利夫医院）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Queen Mary University London（伦敦女王学院）；Barts Health NHS Trust（巴特斯医疗信托）
ETH Zurich（苏黎世联邦理工学院）；Beijing Institute of Technology（北京理工大学）
Despite recent progress of automatic medical image segmentation techniques, fully automatic results usually fail to meet the clinical use and typically require further refinement. In this work, we propose a quality-aware memory network for interactive segmentation of 3D medical images. Provided by user guidance on an arbitrary slice, an interaction network is firstly employed to obtain an initial 2D segmentation. The quality-aware memory network subsequently propagates the initial segmentation estimation bidirectionally over the entire volume. Subsequent refinement based on additional user gui…
Xiamen University（厦门大学）；The University of Hong Kong（香港大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）深圳大数据研究院）
Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to overcome their inherent shortcomings of spatial inductive bias. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer, a 3D transformer for volumetric medical i…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Deep learning has made a remarkable impact in the field of natural image processing over the past decade. Consequently, there is a great deal of interest in replicating this success across unsolved tasks in related domains, such as medical image analysis. Core to medical image analysis is the task of semantic segmentation which enables various clinical workflows. Due to the challenges inherent in manual segmentation, many decades of research have been devoted to discovering extensible, automated, expert-level segmentation techniques. Given the groundbreaking performance demonstrated by recent …
Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医疗机器人研究所）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Key Lab of Computer Vision and Pattern Recognition, SIAT-SenseTime Joint Lab, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院SIAT-商汤联合实验室计算机视觉与模式识别深圳市重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院Paul C. Lauterbur生物医学成像研究中心）；Peng Cheng Laboratory（鹏城实验室）；Pazhou Lab（琶洲实验室）；School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院）
Recent studies have witnessed the effectiveness of 3D convolutions on segmenting volumetric medical images. Compared with the 2D counterparts, 3D convolutions can capture the spatial context in three dimensions. Nevertheless, models employing 3D convolutions introduce more trainable parameters and are more computationally complex, which may lead easily to model overfitting especially for medical applications with limited available training data. This paper aims to improve the effectiveness and efficiency of 3D convolutions by introducing a novel Group Shift Pointwise Convolution (GSP-Conv). GS…
Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning
Mallinckrodt Institute of Radiology（马林克罗德放射医学研究所）；Institute for Informatics, Data Science and Biostatistics（信息学、数据科学与生物统计学研究所）；Department of Radiology, Perelman School of Medicine（放射科，佩尔曼医学院）；Center for AI and Data Science for Integrated Diagnostics (AI2D)（集成诊断人工智能与数据科学中心（AI2D））
Recently deep residual learning with residual units for training very deep neural networks advanced the state-of-the-art performance on 2D image recognition tasks, e.g., object detection and segmentation. However, how to fully leverage contextual representations for recognition tasks from volumetric data has not been well studied, especially in the field of medical image computing, where a majority of image modalities are in volumetric format. In this paper we explore the deep residual learning on the task of volumetric brain segmentation. There are at least two main contributions in our work.…
Technical University of Munich（慕尼黑工业大学）；Ulm University（乌尔姆大学）
The lack of sufficient annotated image data is a common issue in medical image segmentation. For some organs and densities, the annotation may be scarce, leading to poor model training convergence, while other organs have plenty of annotated data. In this work, we present MetaMedSeg, a gradient-based meta-learning algorithm that redefines the meta-learning task for the volumetric medical data with the goal to capture the variety between the slices. We also explore different weighting schemes for gradients aggregation, arguing that different tasks might have different complexity, and hence, con…
Segmentation is a critical step in medical image analysis. Fully Convolutional Networks (FCNs) have emerged as powerful segmentation models achieving state-of-the-art results in various medical image datasets. Network architectures are usually designed manually for a specific segmentation task so applying them to other medical datasets requires extensive experience and time. Moreover, the segmentation requires handling large volumetric data that results in big and complex architectures. Recently, methods that automatically design neural networks for medical image segmentation have been present…
Multi-level Asymmetric Contrastive Learning for Volumetric Medical Image Segmentation Pre-training
Peking University（北京大学）；Peking University Health Science Center（北京大学医学部）；National Biomedical Imaging Center（国家生物医学成像中心）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Georgia Institute of Technology（佐治亚理工学院）；UT Southwestern Medical Center（西南大学达拉斯医学中心）
The Hong Kong Polytechnic University（香港理工大学）；University of Oxford（牛津大学）
Automatic medical image segmentation based on Computed Tomography (CT) has been widely applied for computer-aided surgery as a prerequisite. With the development of deep learning technologies, deep convolutional neural networks (DCNNs) have shown robust performance in automated semantic segmentation of medical images. However, semantic segmentation algorithms based on DCNNs still meet the challenges of feature loss between encoder and decoder, multi-scale object, restricted field of view of filters, and lack of medical image data. This paper proposes a novel algorithm for automated vertebrae s…
Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentation
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（HKUST深圳-香港协同创新研究院）
VinAI Research（VinAI研究院）；University of Arkansas（阿肯色大学）
Medical image segmentation has been so far achieving promising results with Convolutional Neural Networks (CNNs). However, it is arguable that in traditional CNNs, its pooling layer tends to discard important information such as positions. Moreover, CNNs are sensitive to rotation and affine transformation. Capsule network is a data-efficient network design proposed to overcome such limitations by replacing pooling layers with dynamic routing and convolutional strides, which aims to preserve the part-whole relationships. Capsule network has shown a great performance in image recognition and nat…
Deep learning highly relies on the quantity of annotated data. However, the annotations for 3D volumetric medical data require experienced physicians to spend hours or even days for investigation. Self-supervised learning is a potential solution to get rid of the strong requirement of training data by deeply exploiting raw data information. In this paper, we propose a novel self-supervised learning framework for volumetric medical images. Specifically, we propose a context restoration task, i.e., Rubik's cube++, to pre-train 3D neural networks. Different from the existing context-restoration-b…
Yale University（耶鲁大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Contrastive learning (CL) aims to learn useful representation without relying on expert annotations in the context of medical image segmentation. Existing approaches mainly contrast a single positive vector (i.e., an augmentation of the same image) against a set of negatives within the entire remainder of the batch by simply mapping all input features into the same constant vector. Despite the impressive empirical performance, those methods have the following shortcomings: (1) it remains a formidable challenge to prevent the collapsing problems to trivial solutions; and (2) we argue that not a…
Structure and Progress Aware Diffusion for Medical Image Segmentation
School of Artificial Intelligence（人工智能学院）；School of Computer Science and Technology（计算机科学与技术学院）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Security Artificial Intelligence（安徽省安全人工智能重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）
Dynamic Prompt Generation for Interactive 3D Medical Image Segmentation Training
Medical Physics, Department of Diagnostic and Interventional Radiology, Medical Center—University of Freiburg, Faculty of Medicine, University of Freiburg（医学物理，诊断与介入放射科，弗赖堡大学医学中心，医学院，弗赖堡大学）；University of Freiburg（弗赖堡大学）；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Medical image segmentation is one of the most challenging tasks in medical image analysis and has been widely developed for many clinical applications. Most of the existing metrics have been first designed for natural images and then extended to medical images. While object surface plays an important role in medical segmentation and quantitative analysis i.e. analyze brain tumor surface, measure gray matter volume, most of the existing metrics are limited when it comes to analyzing the object surface, especially to tell about surface smoothness or roughness of a given volumetric object or to a…
Insel Data Science Center（因塞尔数据科学中心）；Inselspital, Bern University Hospital（伯尔尼大学医院因塞尔医院）；University of Bern（伯尔尼大学）；Support Center for Advanced Neuroimaging (SCAN)（高级神经成像支持中心（SCAN））；Institute for Diagnostic and Interventional Neuroradiology（诊断与介入神经放射学研究所）
Volumetric image segmentation with convolutional neural networks (CNNs) encounters several challenges, which are specific to medical images. Among these challenges are large volumes of interest, high class imbalances, and difficulties in learning shape representations. To tackle these challenges, we propose to improve over traditional CNN-based volumetric image segmentation through point-wise classification of point clouds. The sparsity of point clouds allows processing of entire image volumes, balancing highly imbalanced segmentation problems, and explicitly learning an anatomical shape. We b…
In recent years, 3D convolutional neural networks have become the dominant approach for volumetric medical image segmentation. However, compared to their 2D counterparts, 3D networks introduce substantially more training parameters and higher requirement for the GPU memory. This has become a major limiting factor for designing and training 3D networks for high-resolution volumetric images. In this work, we propose a novel memory-efficient network architecture for 3D high-resolution image segmentation. The network incorporates both global and local features via a two-stage U-net-based cascaded …
Contextual Embedding Learning to Enhance 2D Networks for Volumetric Image Segmentation
Shenzhen University（深圳大学）；Shenzhen University Medical School（深圳大学医学院）；School of Biomedical Engineering（生物医学工程学院）；Huawei Cloud Computing Technologies Company Limited（华为云计算技术有限公司）；Shenzhen University General Hospital（深圳大学总医院）
Institute of Neurology, UCL（伦敦大学学院神经病学研究所）；School of Biomedical Engineering and Imaging Sciences, Kings College London（伦敦国王学院生物医学工程与成像科学学院）
Deep generative models are rapidly gaining traction in medical imaging. Nonetheless, most generative architectures struggle to capture the underlying probability distributions of volumetric data, exhibit convergence problems, and offer no robust indices of model uncertainty. By comparison, the autoregressive generative model PixelCNN can be extended to volumetric data with relative ease, it readily attempts to learn the true underlying probability distribution and it still admits a Bayesian reformulation that provides a principled framework for reasoning about model uncertainty. Our contributi…
SegVol: Universal and Interactive Volumetric Medical Image Segmentation
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；BAAI（北京智源人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）
Med-Tuning: A New Parameter-Efficient Tuning Framework for Medical Volumetric Segmentation
School of Automation and Electrical Engineering, University of Science and Technology Beijing（北京科技大学自动化与电气工程学院）；Center for Research in Computer Vision, University of Central Florida（中佛罗里达大学计算机视觉研究中心）；School of Computer Science, University of Birmingham（伯明翰大学计算机学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
