In the last decade, convolutional neural networks (ConvNets) have dominated and achieved state-of-the-art performances in a variety of medical imaging applications. However, the performances of ConvNets are still limited by lacking the understanding of long-range spatial relations in an image. The recently proposed Vision Transformer (ViT) for image classification uses a purely self-attention-based model that learns long-range spatial relations to focus on the relevant parts of an image. Nevertheless, ViT emphasizes the low-resolution features because of the consecutive downsamplings, result in a lack of detailed localization information, making it unsuitable for image registration. Recently, several ViT-based image segmentation methods have been combined with ConvNets to improve the recovery of detailed localization information. Inspired by them, we present ViT-V-Net, which bridges ViT and ConvNet to provide volumetric medical image registration. The experimental results presented here demonstrate that the proposed architecture achieves superior performance to several top-performing registration methods.
University of Magdeburg（马格德堡大学）；University Hospital Magdeburg（马格德堡大学医院）
Recent studies demonstrated the eligibility of convolutional neural networks (CNNs) for solving the image registration problem. CNNs enable faster transformation estimation and greater generalization capability needed for better support during medical interventions. Conventional fully-supervised training requires a lot of high-quality ground truth data such as voxel-to-voxel transformations, which typically are attained in a too tedious and error-prone manner. In our work, we use weakly-supervised learning, which optimizes the model indirectly only via segmentation masks that are a more access…
ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究中心）
Image registration of liver dynamic contrast-enhanced computed tomography (DCE-CT) is crucial for diagnosis and image-guided surgical planning of liver cancer. However, intensity variations due to the flow of contrast agents combined with complex spatial motion induced by respiration brings great challenge to existing intensity-based registration methods. To address these problems, we propose a novel structure-aware registration method by incorporating structural information of related organs with segmentation-guided deep registration network. Existing segmentation-guided registration methods …
UT Southwestern Medical Center（德克萨斯大学西南医学中心）；Southern Methodist University（南方卫理公会大学）
Medical image registration is a fundamental and vital task which will affect the efficacy of many downstream clinical tasks. Deep learning (DL)-based deformable image registration (DIR) methods have been investigated, showing state-of-the-art performance. A test time optimization (TTO) technique was proposed to further improve the DL models' performance. Despite the substantial accuracy improvement with this TTO technique, there still remained some regions that exhibited large registration errors even after many TTO iterations. To mitigate this challenge, we firstly identified the reason why t…
Cortical surface registration is a fundamental tool for neuroimaging analysis that has been shown to improve the alignment of functional regions relative to volumetric approaches. Classically, image registration is performed by optimizing a complex objective similarity function, leading to long run times. This contributes to a convention for aligning all data to a global average reference frame that poorly reflects the underlying cortical heterogeneity. In this paper, we propose a novel unsupervised learning-based framework that converts registration to a multi-label classification problem, wh…
Multi-modal image registration is a challenging task that is vital to fuse complementary signals for subsequent analyses. Despite much research into cost functions addressing this challenge, there exist cases in which these are ineffective. In this work, we show that (1) this is true for the registration of in-vivo Drosophila brain volumes visualizing genetically encoded calcium indicators to an nc82 atlas and (2) that machine learning based contrast synthesis can yield improvements. More specifically, the number of subjects for which the registration outright failed was greatly reduced (from …
Shiraz University of Medical Sciences（设拉子医科大学）；University of Pittsburgh School of Medicine（匹兹堡大学医学院）；UPMC Hillman Cancer Center（UPMC希尔曼癌症中心）；Shiraz University of Technology（设拉子理工大学）；Mashhad University of Medical Science（马什哈德医科大学）
In medicine, image registration is vital in image-guided interventions and other clinical applications. However, it is a difficult subject to be addressed which by the advent of machine learning, there have been considerable progress in algorithmic performance has recently been achieved for medical image registration in this area. The implementation of deep neural networks provides an opportunity for some medical applications such as conducting image registration in less time with high accuracy, playing a key role in countering tumors during the operation. The current study presents a comprehe…
Uppsala University（乌普萨拉大学）；Mathematical Institute of the Serbian Academy of Sciences and Arts（塞尔维亚科学院数学研究所）
Intensity-based image registration approaches rely on similarity measures to guide the search for geometric correspondences with high affinity between images. The properties of the used measure are vital for the robustness and accuracy of the registration. In this study a symmetric, intensity interpolation-free, affine registration framework based on a combination of intensity and spatial information is proposed. The excellent performance of the framework is demonstrated on a combination of synthetic tests, recovering known transformations in the presence of noise, and real applications in bio…
Reliable feature point matching is a vital yet challenging process in feature-based image registration. In this paper,a robust feature point matching algorithm called Recovery and Filtering Vertex Trichotomy Matching (RFVTM) is proposed to remove outliers and retain sufficient inliers for remote sensing images. A novel affine invariant descriptor called vertex trichotomy descriptor is proposed on the basis of that geometrical relations between any of vertices and lines are preserved after affine transformations, which is constructed by mapping each vertex into trichotomy sets. The outlier remo…
KAIST（韩国科学技术院）；Seoul National University College of Medicine（首尔大学医学院）；Neuroscience Research Institute, Seoul National University College of Medicine（首尔大学医学院神经科学研究所）；Pohang University of Science and Technology（浦项科技大学）
Volumetric imaging by fluorescence microscopy is often limited by anisotropic spatial resolution from inferior axial resolution compared to the lateral resolution. To address this problem, here we present a deep-learning-enabled unsupervised super-resolution technique that enhances anisotropic images in volumetric fluorescence microscopy. In contrast to the existing deep learning approaches that require matched high-resolution target volume images, our method greatly reduces the effort to put into practice as the training of a network requires as little as a single 3D image stack, without a pr…
We present a parallel distributed-memory algorithm for large deformation diffeomorphic registration of volumetric images that produces large isochoric deformations (locally volume preserving). Image registration is a key technology in medical image analysis. Our algorithm uses a partial differential equation constrained optimal control formulation. Finding the optimal deformation map requires the solution of a highly nonlinear problem that involves pseudo-differential operators, biharmonic operators, and pure advection operators both forward and back- ward in time. A key issue is the time to s…
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
University of Alberta（阿尔伯塔大学）；Mazankowski Alberta Heart Institute（马赞科夫斯基阿尔伯塔心脏研究所）
Purpose: Echocardiography is commonly used as a non-invasive imaging tool in clinical practice for the assessment of cardiac function. However, delineation of the left ventricle is challenging due to the inherent properties of ultrasound imaging, such as the presence of speckle noise and the low signal-to-noise ratio. Methods: We propose a semi-automated segmentation algorithm for the delineation of the left ventricle in temporal 3D echocardiography sequences. The method requires minimal user interaction and relies on a diffeomorphic registration approach. Advantages of the method include no d…
Continuous sPatial-Temporal Deformable Image Registration (CPT-DIR) for motion modelling in radiotherapy: beyond classic voxel-based methods
Center for Proton Therapy, Paul Scherrer Institut（保罗·施特格研究所质子治疗中心）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）；Department of Physics, ETH Zürich（苏黎世联邦理工学院物理系）；Department of Automation, Tsinghua Univeristy（清华大学自动化系）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
Advances in imaging and early cancer detection have increased interest in magnetic resonance (MR) guided focused ultrasound (MRgFUS) technologies for cancer treatment. MRgFUS ablation treatments could reduce surgical risks, preserve organ tissue/function, and improve patient quality of life. However, surgical resection and histological analysis remain the gold standard to assess cancer treatment response. For non-invasive ablation therapies such as MRgFUS, the treatment response must be determined through MR imaging biomarkers. However, current MR biomarkers are inconclusive and have not been …
COph100: A comprehensive fundus image registration dataset from infants constituting the "RIDIRP" database
Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University（首都医科大学附属北京同仁医院北京同仁眼科中心）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Fourier-Net: Fast Image Registration with Band-limited Deformation
School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）；University of Melbourne（墨尔本大学）；University of Cambridge（剑桥大学）；University of Warwick（华威大学）；Institute of Information Computer Engineering, Northeast Forestry University（东北林业大学信息与计算机工程学院）；Alan Turing Institute（阿兰·图灵研究所）
Shanghai Jiao Tong University（上海交通大学）；University of Sydney（悉尼大学）
Dynamic medical imaging is usually limited in application due to the large radiation doses and longer image scanning and reconstruction times. Existing methods attempt to reduce the dynamic sequence by interpolating the volumes between the acquired image volumes. However, these methods are limited to either 2D images and/or are unable to support large variations in the motion between the image volume sequences. In this paper, we present a spatiotemporal volumetric interpolation network (SVIN) designed for 4D dynamic medical images. SVIN introduces dual networks: first is the spatiotemporal mot…
Robustness and Diagnostic Performance of Super-Resolution Fetal Brain MRI
Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia（卢布尔雅那大学医学学院）；University Medical Centre Ljubljana, Ljubljana, Slovenia（卢布尔雅那大学医学中心）；Faculty of Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia（卢布尔雅那大学电气工程学院）
IIT Kharagpur（印度理工学院克勒格布尔分校）；IIT Bombay（印度理工学院孟买分校）；IIT Madras（印度理工学院马德拉斯分校）；Philips Research Philips Healthcare（飞利浦研究 飞利浦医疗保健）
In the medical field, landmark detection in MRI plays an important role in reducing medical technician efforts in tasks like scan planning, image registration, etc. First, 88 landmarks spread across the brain anatomy in the three respective views -- sagittal, coronal, and axial are manually annotated, later guidelines from the expert clinical technicians are taken sub-anatomy-wise, for better localization of the existing landmarks, in order to identify and locate the important atlas landmarks even in oblique scans. To overcome limited data availability, we implement realistic data augmentation…
A method for supervoxel-wise association studies of age and other non-imaging variables from coronary computed tomography angiograms
Radiology, Department of Surgical Sciences, Uppsala University（乌普萨拉大学放射科）；Antaros Medical（Antaros医疗）；Department of Molecular and Clinical Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg（哥德堡大学医学院分子与临床医学系）
Multi-atlas segmentation (MAS) is a popular image segmentation technique for medical images. In this work, we improve the performance of MAS by correcting registration errors before label fusion. Specifically, we use a volumetric displacement field to refine registrations based on image anatomical appearance and predicted labels. We show the influence of the initial spatial alignment as well as the beneficial effect of using label information for MAS performance. Experiments demonstrate that the proposed refinement approach improves MAS performance on a 3D magnetic resonance dataset of the kne…
Universidad Politécnica de Madrid（马德里理工大学）；Biomedical Research Networking Center in Bioengineering, Biomaterials and Nanomedicine (CIBER-BBN)（生物医学研究网络中心（生物工程、生物材料与纳米医学））；King’s College London（伦敦国王学院）
Magnetic resonance imaging of whole fetal body and placenta is limited by different sources of motion affecting the womb. Usual scanning techniques employ single-shot multi-slice sequences where anatomical information in different slices may be subject to different deformations, contrast variations or artifacts. Volumetric reconstruction formulations have been proposed to correct for these factors, but they must accommodate a non-homogeneous and non-isotropic sampling, so regularization becomes necessary. Thus, in this paper we propose a deep generative prior for robust volumetric reconstructi…
MIT（麻省理工学院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；University College London（伦敦大学学院）；Massachusetts General Hospital（麻省总医院）
Volumetric reconstruction of fetal brains from multiple stacks of MR slices, acquired in the presence of almost unpredictable and often severe subject motion, is a challenging task that is highly sensitive to the initialization of slice-to-volume transformations. We propose a novel slice-to-volume registration method using Transformers trained on synthetically transformed data, which model multiple stacks of MR slices as a sequence. With the attention mechanism, our model automatically detects the relevance between slices and predicts the transformation of one slice using information from othe…
Medical Imaging Research Center, UZ Leuven（UZ Leuven医学成像研究中心）；Department of Electrical Engineering, ESAT/PSI, KU Leuven（KU Leuven电气工程系（ESAT/PSI））；Center for Craniofacial and Dental Genetics, Department of Oral and Craniofacial Sciences, University of Pittsburgh（匹兹堡大学口腔与颅面科学系颅面和牙科遗传学中心）；Department of Computing, Imperial College London（帝国理工学院计算机系）；IDSIA, USI Lugano（卢加诺大学IDSIA研究所）；Twitter（推特公司）；Department of Human Genetics, KU Leuven（KU Leuven人类遗传学系）
Registration is an essential tool in image analysis. Deep learning based alternatives have recently become popular, achieving competitive performance at a faster speed. However, many contemporary techniques are limited to volumetric representations, despite increased popularity of 3D surface and shape data in medical image analysis. We propose a one-step registration model for 3D surfaces that internalises a lower dimensional probabilistic deformation model (PDM) using conditional variational autoencoders (CVAE). The deformations are constrained to be diffeomorphic using an exponentiation laye…
Volumetric (3d) images are acquired for many scientific and biomedical purposes using imaging methods such as serial section microscopy, CT scans, and MRI. A frequent step in the analysis and reconstruction of such data is the alignment and registration of images that were acquired in succession along a spatial or temporal dimension. For example, in serial section electron microscopy, individual 2d sections are imaged via electron microscopy and then must be aligned to one another in order to produce a coherent 3d volume. State of the art approaches find image correspondences derived from patc…
Institute of Parallel and Distributed Systems, University of Stuttgart（斯图加特大学并行与分布式系统研究所）
Deformable image registration is a fundamental task in medical imaging. Due to the large computational complexity of deformable registration of volumetric images, conventional iterative methods usually face the tradeoff between the registration accuracy and the computation time in practice. In order to boost the registration performance in both accuracy and runtime, we propose a fast convolutional neural network. Specially, to efficiently utilize the memory resources and enlarge the model capacity, we adopt additive forwarding instead of channel concatenation and deepen the network in each enc…
Columbia University（哥伦比亚大学）；Columbia University Medical Center（哥伦比亚大学医学中心）；University of Iowa（爱荷华大学）
Unsupervised learning-based medical image registration approaches have witnessed rapid development in recent years. We propose to revisit a commonly ignored while simple and well-established principle: recursive refinement of deformation vector fields across scales. We introduce a recursive refinement network (RRN) for unsupervised medical image registration, to extract multi-scale features, construct normalized local cost correlation volume and recursively refine volumetric deformation vector fields. RRN achieves state of the art performance for 3D registration of expiratory-inspiratory pairs…
Anatomically plausible image registration often requires volumetric preservation. Previous approaches to incompressible image registration have exploited relaxed constraints, ad hoc optimisation methods or practically intractable computational schemes. Divergence-free velocity fields have been used to achieve incompressibility in the continuous domain, although, after discretisation, no guarantees have been provided. In this paper, we introduce stationary velocity fields (SVFs) parameterised by divergence-conforming B-splines in the context of image registration. We demonstrate that sparse lin…
Volumetric cell segmentation in fluorescence microscopy images is important to study a wide variety of cellular processes. Applications range from the analysis of cancer cells to behavioral studies of cells in the embryonic stage. Like in other computer vision fields, most recent methods use either large convolutional neural networks (CNNs) or vision transformer models (ViTs). Since the number of available 3D microscopy images is typically limited in applications, we take a different approach and introduce a small CNN for volumetric cell segmentation. Compared to previous CNN models for cell s…
Johns Hopkins Medical Institutes（约翰·霍普金斯医学院）；Johns Hopkins University（约翰·霍普金斯大学）；NVIDIA Corporation（英伟达公司）；Duke University Medical Center（杜克大学医学中心）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehensi…
University of Oxford（牛津大学）；University College London（伦敦大学学院）；InstaDeep（InstaDeep公司）；City University of Hong Kong（香港城市大学）；Radboud University Nijmegen Medical Centre（拉德堡德大学奈梅亨医学中心）
The ability to adapt medical image segmentation networks for a novel class such as an unseen anatomical or pathological structure, when only a few labelled examples of this class are available from local healthcare providers, is sought-after. This potentially addresses two widely recognised limitations in deploying modern deep learning models to clinical practice, expertise-and-labour-intensive labelling and cross-institution generalisation. This work presents the first 3D few-shot interclass segmentation network for medical images, using a labelled multi-institution dataset from prostate canc…
Purpose: Diffeomorphic image registration is essential in many medical imaging applications. Several registration algorithms of such type have been proposed, but primarily for intra-contrast alignment. Currently, efficient inter-modal/contrast diffeomorphic registration, which is vital in numerous applications, remains a challenging task. Methods: We proposed a novel inter-modal/contrast registration algorithm that leverages Robust PaTch-based cOrrelation Ratio (RaPTOR) metric to allow inter-modal/contrast image alignment and bandlimited geodesic shooting demonstrated in Fourier Approximated L…
Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentation
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（HKUST深圳-香港协同创新研究院）
A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School（哈佛医学院麻省总医院A.A.马丁诺斯生物医学成像中心）；Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
Non-rigid cortical registration is an important and challenging task due to the geometric complexity of the human cortex and the high degree of inter-subject variability. A conventional solution is to use a spherical representation of surface properties and perform registration by aligning cortical folding patterns in that space. This strategy produces accurate spatial alignment but often requires a high computational cost. Recently, convolutional neural networks (CNNs) have demonstrated the potential to dramatically speed up volumetric registration. However, due to distortions introduced by p…
Investigating the impact of kernel harmonization and deformable registration on inspiratory and expiratory chest CT images for people with COPD
Vanderbilt University（范德堡大学）；Insitro；Vanderbilt University Medical Center（范德堡大学医学中心）；Department of Veterans Affairs Medical Center（退伍军人事务部医疗中心）；Vanderbilt University Institute of Imaging Science（范德堡大学成像科学研究所）
针对 COPD 患者吸呼气 CT 因重建核差异导致定量分析误差的问题，提出两阶段流程：先用循环 GAN 进行核 harmonization，再进行可变形配准，显著降低肺气肿测量不一致性并提高配准精度。
X-Ray imaging is quick, cheap and useful for front-line care assessment and intra-operative real-time imaging (e.g., C-Arm Fluoroscopy). However, it suffers from projective information loss and lacks vital volumetric information on which many essential diagnostic biomarkers are based on. In this paper we explore probabilistic methods to reconstruct 3D volumetric images from 2D imaging modalities and measure the models' performance and confidence. We show our models' performance on large connected structures and we test for limitations regarding fine structures and image domain sensitivity. We …
