Recent works have shown that the computational efficiency of 3D medical image (e.g. CT and MRI) segmentation can be impressively improved by dynamic inference based on slice-wise complexity. As a pioneering work, a dynamic architecture network for medical volumetric segmentation (i.e. Med-DANet) has achieved a favorable accuracy and efficiency trade-off by dynamically selecting a suitable 2D candidate model from the pre-defined model bank for different slices. However, the issues of incomplete data analysis, high training costs, and the two-stage pipeline in Med-DANet require further improvement. To this end, this paper further explores a unified formulation of the dynamic inference framework from the perspective of both the data itself and the model structure. For each slice of the input volume, our proposed method dynamically selects an important foreground region for segmentation based on the policy generated by our Decision Network and Crop Position Network. Besides, we propose to insert a stage-wise quantization selector to the employed segmentation model (e.g. U-Net) for dynamic architecture adapting. Extensive experiments on BraTS 2019 and 2020 show that our method achieves comparable or better performance than previous state-of-the-art methods with much less model complexity. Compared with previous methods Med-DANet and TransBTS with dynamic and static architecture respectively, our framework improves the model efficiency by up to nearly 4.1 and 17.3 times with comparable segmentation results on BraTS 2019.
University of Science and Technology Beijing（北京科技大学）；University of Central Florida（中佛罗里达大学）
For 3D medical image (e.g. CT and MRI) segmentation, the difficulty of segmenting each slice in a clinical case varies greatly. Previous research on volumetric medical image segmentation in a slice-by-slice manner conventionally use the identical 2D deep neural network to segment all the slices of the same case, ignoring the data heterogeneity among image slices. In this paper, we focus on multi-modal 3D MRI brain tumor segmentation and propose a dynamic architecture network named Med-DANet based on adaptive model selection to achieve effective accuracy and efficiency trade-off. For each slice…
Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）
Dynamic medical imaging is usually limited in application due to the large radiation doses and longer image scanning and reconstruction times. Existing methods attempt to reduce the dynamic sequence by interpolating the volumes between the acquired image volumes. However, these methods are limited to either 2D images and/or are unable to support large variations in the motion between the image volume sequences. In this paper, we present a spatiotemporal volumetric interpolation network (SVIN) designed for 4D dynamic medical images. SVIN introduces dual networks: first is the spatiotemporal mot…
Beihang University（北京航空航天大学）；Hefei Innovation Research Institute, Beihang University（北京航空航天大学合肥创新研究院）；Beijing Advanced Innovation Centre for Biomedical Engineering（北京生物医学工程先进创新中心）；Beijing Advanced Innovation Centre for Big Data-Based Precision Medicine（北京大数据精准医疗先进创新中心）
Recently, deep convolutional neural networks have achieved great success for medical image segmentation. However, unlike segmentation of natural images, most medical images such as MRI and CT are volumetric data. In order to make full use of volumetric information, 3D CNNs are widely used. However, 3D CNNs suffer from higher inference time and computation cost, which hinders their further clinical applications. Additionally, with the increased number of parameters, the risk of overfitting is higher, especially for medical images where data and annotations are expensive to acquire. To issue thi…
SAM3D: Segment Anything Model in Volumetric Medical Images
University of Arkansas（阿肯色大学）；University of Science, Vietnam National University（越南国家大学科学大学）；John von Neumann Institute, Vietnam National University（越南国家大学约翰·冯·诺依曼研究所）；West Virginia University（西弗吉尼亚大学）；University of Arkansas for Medical Sciences（阿肯色医科大学）
University of Science and Technology Beijing（北京科技大学）；University of Central Florida（中佛罗里达大学）
Transformer, benefiting from global (long-range) information modeling using self-attention mechanism, has been successful in natural language processing and computer vision recently. Convolutional Neural Networks, capable of capturing local features, are difficult to model explicit long-distance dependencies from global feature space. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we present the further attempt to exploit Transformer in 3D CNN for 3D medical image volumetric segmentation and propose a…
CitAI Research Centre（CitAI研究中心）；City, University of London（伦敦城市大学）；BioMedIA（生物医学影像分析组）；Imperial College（帝国理工学院）
U-Net has been the go-to architecture for medical image segmentation tasks, however computational challenges arise when extending the U-Net architecture to 3D images. We propose the Implicit U-Net architecture that adapts the efficient Implicit Representation paradigm to supervised image segmentation tasks. By combining a convolutional feature extractor with an implicit localization network, our implicit U-Net has 40% less parameters than the equivalent U-Net. Moreover, we propose training and inference procedures to capitalize sparse predictions. When comparing to an equivalent fully convolut…
LMU München（慕尼黑大学）；Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Fully Convolutional Neural Networks (F-CNNs) achieve state-of-the-art performance for image segmentation in medical imaging. Recently, squeeze and excitation (SE) modules and variations thereof have been introduced to recalibrate feature maps channel- and spatial-wise, which can boost performance while only minimally increasing model complexity. So far, the development of SE has focused on 2D images. In this paper, we propose `Project & Excite' (PE) modules that base upon the ideas of SE and extend them to operating on 3D volumetric images. `Project & Excite' does not perform global average po…
Korea Advanced Institute of Science and Technology（韩国高等科学技术研究院）；National Cancer Center（国家癌症中心）
Volumetric images from Magnetic Resonance Imaging (MRI) provide invaluable information in preoperative staging of rectal cancer. Above all, accurate preoperative discrimination between T2 and T3 stages is arguably both the most challenging and clinically significant task for rectal cancer treatment, as chemo-radiotherapy is usually recommended to patients with T3 (or greater) stage cancer. In this study, we present a volumetric convolutional neural network to accurately discriminate T2 from T3 stage rectal cancer with rectal MR volumes. Specifically, we propose 1) a custom ResNet-based volume …
Segmentation is a fundamental task in medical image analysis. The clinical interest is often to measure the volume of a structure. To evaluate and compare segmentation methods, the similarity between a segmentation and a predefined ground truth is measured using metrics such as the Dice score. Recent segmentation methods based on convolutional neural networks use a differentiable surrogate of the Dice score, such as soft Dice, explicitly as the loss function during the learning phase. Even though this approach leads to improved Dice scores, we find that, both theoretically and empirically on f…
SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation
Shanghai AI Laboratory（上海人工智能实验室）；Xiamen University（厦门大学）；East China Normal University（华东师范大学）；University of Cambridge（剑桥大学）；Stanford University（斯坦福大学）
This paper discusses current methods and trends for 3D bounding box detection in volumetric medical image data. For this purpose, an overview of relevant papers from recent years is given. 2D and 3D implementations are discussed and compared. Multiple identified approaches for localizing anatomical structures are presented. The results show that most research recently focuses on Deep Learning methods, such as Convolutional Neural Networks vs. methods with manual feature engineering, e.g. Random-Regression-Forests. An overview of bounding box detection options is presented and helps researchers…
Stanford University（斯坦福大学）；National Institute of Standards and Technology (NIST)（美国国家标准与技术研究院）
Segmentation of additive manufacturing (AM) defects in X-ray Computed Tomography (XCT) images is challenging, due to the poor contrast, small sizes and variation in appearance of defects. Automatic segmentation can, however, provide quality control for additive manufacturing. Over recent years, three-dimensional convolutional neural networks (3D CNNs) have performed well in the volumetric segmentation of medical images. In this work, we leverage techniques from the medical imaging domain and propose training a 3D U-Net model to automatically segment defects in XCT images of AM samples. This wo…
The University of Tokyo Hospital（东京大学医院）；Chiba University（千叶大学）
From birth to death, we all experience surprisingly ubiquitous changes over time due to aging. If we can predict aging in the digital domain, that is, the digital twin of the human body, we would be able to detect lesions in their very early stages, thereby enhancing the quality of life and extending the life span. We observed that none of the previously developed digital twins of the adult human body explicitly trained longitudinal conversion rules between volumetric medical images with deep generative models, potentially resulting in poor prediction performance of, for example, ventricular v…
SegVol: Universal and Interactive Volumetric Medical Image Segmentation
School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；BAAI（北京智源人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）
College of Biomedical Engineering, Rangsit University（兰实大学生物医学工程学院）
Automated medical image segmentation is a priority research area for computational methods. In particular, detection of cancerous tumors represents a current challenge in this area with potential for real-world impact. This paper describes a method developed in response to the 2019 Kidney Tumor Segmentation Challenge (KiTS19). Axial computed tomography (CT) scans from 210 kidney cancer patients were used to develop and evaluate this automatic segmentation method based on a logical ensemble of fully-convolutional network (FCN) architectures, followed by volumetric validation. Data was pre-proce…
University of Pittsburgh（匹兹堡大学）；University of Notre Dame（圣母大学）
Supervised deep learning needs a large amount of labeled data to achieve high performance. However, in medical imaging analysis, each site may only have a limited amount of data and labels, which makes learning ineffective. Federated learning (FL) can help in this regard by learning a shared model while keeping training data local for privacy. Traditional FL requires fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to high labeling cost and the requirement of expertise. Contrastive learning (CL), as a self-supervised learning approach, can effectivel…
Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Klinikum Grosshadern（大哈登医院）；Ludwig-Maximilians-Universität München（慕尼黑大学）
Convolutional Neural Networks (CNNs) have been recently employed to solve problems from both the computer vision and medical image analysis fields. Despite their popularity, most approaches are only able to process 2D images while most medical data used in clinical practice consists of 3D volumes. In this work we propose an approach to 3D image segmentation based on a volumetric, fully convolutional, neural network. Our CNN is trained end-to-end on MRI volumes depicting prostate, and learns to predict segmentation for the whole volume at once. We introduce a novel objective function, that we o…
SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
针对现有 volumetric 医学图像分割模型任务专属、泛化性差的问题，提出通用分割模型 SAM-Med3D，基于大规模 3D 医学数据集两阶段训练，在多场景下展现优异性能与下游应用潜力。
Medical image classification is one of the most critical problems in the image recognition area. One of the major challenges in this field is the scarcity of labelled training data. Additionally, there is often class imbalance in datasets as some cases are very rare to happen. As a result, accuracy in classification task is normally low. Deep Learning models, in particular, show promising results on image segmentation and classification problems, but they require very large datasets for training. Therefore, there is a need to generate more of synthetic samples from the same distribution. Previ…
VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
MedVisAI Lab Department of ECE Northwestern University（MedVisAI实验室 电子工程系 西北大学）；Institute for Infocomm Research (I 2 R) A*STAR, Singapore（信息与通信研究所（I 2 R）A*STAR，新加坡）；MedVisAI Lab Lee Kong Chian School of Medicine, Nanyang Technological University（MedVisAI实验室 李科田医学院，南洋理工大学）
University of Notre Dame（圣母大学）；University of Pittsburgh（匹兹堡大学）；Guangdong Provincial People’s Hospital（广东省人民医院）
The success of deep learning heavily depends on the availability of large labeled training sets. However, it is hard to get large labeled datasets in medical image domain because of the strict privacy concern and costly labeling efforts. Contrastive learning, an unsupervised learning technique, has been proved powerful in learning image-level representations from unlabeled data. The learned encoder can then be transferred or fine-tuned to improve the performance of downstream tasks with limited labels. A critical step in contrastive learning is the generation of contrastive data pairs, which i…
ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, USTC Suzhou Institute for Advanced Research（医学影像、机器人、分析计算与学习（MIRACLE）实验室，YRD-RIGHT，中国科学技术大学苏州研究院）；Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology（江苏省多模态数字孪生技术重点实验室）；Biomedical Basic Research Center (BBRC) of Jiangsu Province（江苏省生物医学基础研究中心）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；Anhui IFLYTEK CO., Ltd（安徽科大讯飞股份有限公司）；School of Medicine, Stanford University（医学院，斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, Hefei, Anhui, China（安徽省精密与智能化学重点实验室，合肥，安徽，中国）
Witnessed the development of deep learning, increasing number of studies try to build computer aided diagnosis systems for 3D volumetric medical data. However, as the annotations of 3D medical data are difficult to acquire, the number of annotated 3D medical images is often not enough to well train the deep learning networks. The self-supervised learning deeply exploiting the information of raw data is one of the potential solutions to loose the requirement of training data. In this paper, we propose a self-supervised learning framework for the volumetric medical images. A novel proxy task, i.…
Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation
Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)（金在哲人工智能研究生院，韩国科学技术院（KAIST））；Yonsei University College of Medicine（延世大学医学院）；Seoul National University College of Medicine（首尔大学医学院）；Seoul National University Hospital（首尔大学医院）；Chung-Ang University College of Medicine（中央大学医学院）；Chungnam National University College of Medicine（忠南大学医学院）
QUIZ: An Arbitrary Volumetric Point Matching Method for Medical Image Registration
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Guangdong Medical University（广东医科大学）
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
SegQC: a segmentation network-based framework for multi-metric segmentation quality control and segmentation error detection in volumetric medical images
University of Haifa（海法大学）；Tel Aviv Sourasky Medical Center（特拉维夫苏拉斯基医疗中心）；Sackler Faculty of Medicine, Tel Aviv University（特拉维夫大学萨克勒医学院）；Tel Aviv University（特拉维夫大学）；Sagol Brain Institute（萨戈尔脑研究所）；Sagol School of Neuroscience, Tel Aviv University（特拉维夫大学萨戈尔神经科学学院）；School of Computer Science and Engineering, The Hebrew University of Jerusalem（希伯来大学计算机科学与工程学院）；The Hebrew University of Jerusalem（希伯来大学）；Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem（希伯来大学埃德蒙与莉莉·萨夫拉脑科学中心）
Robust Curve Detection in Volumetric Medical Imaging via Attraction Field
IRA-Labs Ltd（IRA-Labs有限公司）；Kharkevich Institute for Information Transmission Problems（哈尔科维奇信息传输问题研究所）；Lomonosov Moscow State University（莫斯科罗蒙诺索夫国立大学）；AUMI AI Limited（AUMI人工智能有限公司）
LMU München（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Deep neural networks enable highly accurate image segmentation, but require large amounts of manually annotated data for supervised training. Few-shot learning aims to address this shortcoming by learning a new class from a few annotated support examples. We introduce, a novel few-shot framework, for the segmentation of volumetric medical images with only a few annotated slices. Compared to other related works in computer vision, the major challenges are the absence of pre-trained networks and the volumetric nature of medical scans. We address these challenges by proposing a new architecture f…
NVIDIA（英伟达）；Technische Universität München（慕尼黑工业大学）；Ludwig-Maximilian Universität München（慕尼黑大学）
Deep learning has been recently applied to a multitude of computer vision and medical image analysis problems. Although recent research efforts have improved the state of the art, most of the methods cannot be easily accessed, compared or used by either researchers or the general public. Researchers often publish their code and trained models on the internet, but this does not always enable these approaches to be easily used or integrated in stand-alone applications and existing workflows. In this paper we propose a framework which allows easy deployment and access of deep learning methods for…
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Weakly supervised segmentation is an important problem in medical image analysis due to the high cost of pixelwise annotation. Prior methods, while often focusing on weak labels of 2D images, exploit few structural cues of volumetric medical images. To address this, we propose a novel weakly-supervised segmentation strategy capable of better capturing 3D shape prior in both model prediction and learning. Our main idea is to extract a self-taught shape representation by leveraging weak labels, and then integrate this representation into segmentation prediction for shape refinement. To this end,…
Case Western Reserve University（凯斯西储大学）；IBM Almaden Research Center（IBM阿尔马登研究中心）
We explore a solution for learning disease signatures from weakly, yet easily obtainable, annotated volumetric medical imaging data by analyzing 3D volumes as a sequence of 2D images. We demonstrate the performance of our solution in the detection of emphysema in lung cancer screening low-dose CT images. Our approach utilizes convolutional long short-term memory (LSTM) to "scan" sequentially through an imaging volume for the presence of disease in a portion of scanned region. This framework allowed effective learning given only volumetric images and binary disease labels, thus enabling trainin…
Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2
Department of Electrical and Computer Engineering, Duke University（电气与计算机工程系，杜克大学）；Department of Biostatistics and Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Biostatistics and Bioinformatics, Radiology, Electrical and Computer Engineering, and Computer Science, Duke University（生物统计学与生物信息学系，放射学，电气与计算机工程，和计算机科学，杜克大学）
本文提出 SLM-SAM 2 架构，通过分离的短期和长期记忆库及注意力模块解决 SAM 2 在体积医学图像标注中的错误传播问题，在四个数据集上显著提升了分割精度并减少了 60.575% 的掩码校正时间。
In medical image segmentation, supervised deep networks' success comes at the cost of requiring abundant labeled data. While asking domain experts to annotate only one or a few of the cohort's images is feasible, annotating all available images is impractical. This issue is further exacerbated when pre-trained deep networks are exposed to a new image dataset from an unfamiliar distribution. Using available open-source data for ad-hoc transfer learning or hand-tuned techniques for data augmentation only provides suboptimal solutions. Motivated by atlas-based segmentation, we propose a novel vol…
LHU-Net: a Lean Hybrid U-Net for Cost-efficient, High-performance Volumetric Segmentation
Faculty of Informatics and Data Science, University of Regensburg（信息学与数据科学学院，莱茵河畔罗特魏尔大学）；Faculty of Electrical Engineering and Information Technology, RWTH Aachen University（电气工程与信息科技学院，亚琛工业大学）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）
Aligning Fetal Anatomy with Kinematic Tree Log-Euclidean PolyRigid Transforms
Computer Science and Artificial Intelligence Lab, MIT, Cambridge, USA（麻省理工学院计算机科学与人工智能实验室）；California Institute of Technology, Pasadena, USA（加州理工学院）；Boston Children’s Hospital and Harvard Medical School, Boston, USA（哈佛医学院波士顿儿童医院）；The Hebrew University of Jerusalem, Jerusalem, Israel（耶路撒冷希伯来大学）
Deep learning algorithms, in particular 2D and 3D fully convolutional neural networks (FCNs), have rapidly become the mainstream methodology for volumetric medical image segmentation. However, 2D convolutions cannot fully leverage the rich spatial information along the third axis, while 3D convolutions suffer from the demanding computation and high GPU memory consumption. In this paper, we propose to automatically search the network architecture tailoring to volumetric medical image segmentation problem. Concretely, we formulate the structure learning as differentiable neural architecture sear…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学研究所）
In the last decade, convolutional neural networks (ConvNets) have dominated and achieved state-of-the-art performances in a variety of medical imaging applications. However, the performances of ConvNets are still limited by lacking the understanding of long-range spatial relations in an image. The recently proposed Vision Transformer (ViT) for image classification uses a purely self-attention-based model that learns long-range spatial relations to focus on the relevant parts of an image. Nevertheless, ViT emphasizes the low-resolution features because of the consecutive downsamplings, result i…
