The recent application of deep learning technologies in medical image registration has exponentially decreased the registration time and gradually increased registration accuracy when compared to their traditional counterparts. Most of the learning-based registration approaches considers this task as a one directional problem. As a result, only correspondence from the moving image to the target image is considered. However, in some medical procedures bidirectional registration is required to be performed. Unlike other learning-based registration, we propose a registration framework with inverse consistency. The proposed method simultaneously learns forward transformation and backward transformation in an unsupervised manner. We perform training and testing of the method on the publicly available LPBA40 MRI dataset and demonstrate strong performance than baseline registration methods.
School of Computer Science, University of Birmingham（伯明翰大学计算机学院）；Department of Computing and Mathematics, Manchester Metropolitan University（曼彻斯特城市大学计算与数学系）
Johns Hopkins University（约翰斯·霍普金斯大学）；National Institute on Aging（国家老龄化研究所）；National Institute of Health（美国国立卫生研究院）
The majority of deep learning (DL) based deformable image registration methods use convolutional neural networks (CNNs) to estimate displacement fields from pairs of moving and fixed images. This, however, requires the convolutional kernels in the CNN to not only extract intensity features from the inputs but also understand image coordinate systems. We argue that the latter task is challenging for traditional CNNs, limiting their performance in registration tasks. To tackle this problem, we first introduce Coordinate Translator, a differentiable module that identifies matched features between…
Images taken at different times or positions undergo transformations such as rotation, scaling, skewing, and more. The process of aligning different images which have undergone transformations can be done via registration. Registration is desirable when analyzing time-series data for tracking, averaging, or differential diagnoses of diseases. Efficient registration methods exist for rigid (including linear or affine) transformations; however, for non-rigid (also known as non-affine) transformations, current methods are computationally expensive and time-consuming. In this report, I will explor…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Medical Center（峨山医学中心）
Medical image registration is one of the key processing steps for biomedical image analysis such as cancer diagnosis. Recently, deep learning based supervised and unsupervised image registration methods have been extensively studied due to its excellent performance in spite of ultra-fast computational time compared to the classical approaches. In this paper, we present a novel unsupervised medical image registration method that trains deep neural network for deformable registration of 3D volumes using a cycle-consistency. Thanks to the cycle consistency, the proposed deep neural networks can t…
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a reg…
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Radboud University Medical Center（拉德堡德大学医学中心）
Deep-learning-based registration methods emerged as a fast alternative to conventional registration methods. However, these methods often still cannot achieve the same performance as conventional registration methods because they are either limited to small deformation or they fail to handle a superposition of large and small deformations without producing implausible deformation fields with foldings inside. In this paper, we identify important strategies of conventional registration methods for lung registration and successfully developed the deep-learning counterpart. We employ a Gaussian-py…
Georgia Institute of Technology（佐治亚理工学院）；Massachusetts Institute of Technology（麻省理工学院）；National Institute of Mental Health（美国国家精神卫生研究所）；The Chinese University of Hong Kong（香港中文大学）
Image registration has been widely studied over the past several decades, with numerous applications in science, engineering and medicine. Most of the conventional mathematical models for large deformation image registration rely on prescribed landmarks, which usually require tedious manual labeling and are prone to error. In recent years, there has been a surge of interest in the use of machine learning for image registration. In this paper, we develop a novel method for large deformation image registration by a fusion of quasiconformal theory and convolutional neural network (CNN). More spec…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；National Institutes of Health（美国国立卫生研究院）；National Cancer Institute（美国国家癌症研究所）
Prostate cancer biopsy benefits from accurate fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images. In the past few years, convolutional neural networks (CNNs) have been proved powerful in extracting image features crucial for image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are quite limited in its ability to understand spatial correspondence between features, a task in which the self-attention mechanism excels. This paper aims to develop a self-attention mechanism specifically for cross-modal image registrati…
Westfälische Hochschule University of Applied Sciences（威斯特伐利亚应用科学大学）
Central venous catheters (CVC) are commonly inserted into the large veins of the neck, e.g. the internal jugular vein (IJV). CVC insertion may cause serious complications like misplacement into an artery or perforation of cervical vessels. Placing a CVC under sonographic guidance is an appropriate method to reduce such adverse events, if anatomical landmarks like venous and arterial vessels can be detected reliably. This task shall be solved by registration of patient individual images vs. an anatomically labelled reference image. In this work, a linear, affine transformation is performed on c…
University of Kansas（堪萨斯大学）；Rochester Institute of Technology（罗切斯特理工学院）
Patient-specific left ventricle (LV) myocardial models have the potential to be used in a variety of clinical scenarios for improved diagnosis and treatment plans. Cine cardiac magnetic resonance (MR) imaging provides high resolution images to reconstruct patient-specific geometric models of the LV myocardium. With the advent of deep learning, accurate segmentation of cardiac chambers from cine cardiac MR images and unsupervised learning for image registration for cardiac motion estimation on a large number of image datasets is attainable. Here, we propose a deep leaning-based framework for th…
Medical image registration is a fundamental and critical task in medical image analysis. With the rapid development of deep learning, convolutional neural networks (CNN) have dominated the medical image registration field. Due to the disadvantage of the local receptive field of CNN, some recent registration methods have focused on using transformers for non-local registration. However, the standard Transformer has a vast number of parameters and high computational complexity, which causes Transformer can only be applied at the bottom of the registration models. As a result, only coarse informa…
K L University（K L大学）；MVSR Engineering College（MVSR工程学院）
Image registration is a classic problem of computer vision with several applications across areas like defence, remote sensing, medicine etc. Feature based image registration methods traditionally used hand-crafted feature extraction algorithms, which detect key points in an image and describe them using a region around the point. Such features are matched using a threshold either on distances or ratio of distances computed between the feature descriptors. Evolution of deep learning, in particular convolution neural networks, has enabled researchers to address several problems of vision such a…
Image registration is a key component of various image processing operations which involve the analysis of different image data sets. Automatic image registration domains have witnessed the application of many intelligent methodologies over the past decade; however inability to properly model object shape as well as contextual information had limited the attainable accuracy. In this paper, we propose a framework for accurate feature shape modeling and adaptive resampling using advanced techniques such as Vector Machines, Cellular Neural Network (CNN), SIFT, coreset, and Cellular Automata. CNN …
UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Pennsylvania（宾夕法尼亚大学）
Registration of images with pathologies is challenging due to tissue appearance changes and missing correspondences caused by the pathologies. Moreover, mass effects as observed for brain tumors may displace tissue, creating larger deformations over time than what is observed in a healthy brain. Deep learning models have successfully been applied to image registration to offer dramatic speed up and to use surrogate information (e.g., segmentations) during training. However, existing approaches focus on learning registration models using images from healthy patients. They are therefore not desi…
University of Magdeburg（马格德堡大学）；University Hospital Magdeburg（马格德堡大学医院）
Recent studies demonstrated the eligibility of convolutional neural networks (CNNs) for solving the image registration problem. CNNs enable faster transformation estimation and greater generalization capability needed for better support during medical interventions. Conventional fully-supervised training requires a lot of high-quality ground truth data such as voxel-to-voxel transformations, which typically are attained in a too tedious and error-prone manner. In our work, we use weakly-supervised learning, which optimizes the model indirectly only via segmentation masks that are a more access…
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；School of Computing, University of Utah（犹他大学计算学院）
Spatial transformations are enablers in a variety of medical image analysis applications that entail aligning images to a common coordinate systems. Population analysis of such transformations is expected to capture the underlying image and shape variations, and hence these transformations are required to produce anatomically feasible correspondences. This is usually enforced through some smoothness-based generic regularization on deformation field. Alternatively, population-based regularization has been shown to produce anatomically accurate correspondences in cases where anatomically unaware…
A Symmetric Dynamic Learning Framework for Diffeomorphic Medical Image Registration
Xiangtan University（湘潭大学）；University of Strathclyde（思克莱德大学）；Nankai University（南开大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Manchester（曼彻斯特大学）
Minutia-based palmprint recognition systems has got lots of interest in last two decades. Due to the large number of minutiae in a palmprint, approximately 1000 minutiae, the matching process is time consuming which makes it unpractical for real time applications. One way to address this issue is aligning all palmprint images to a reference image and bringing them to a same coordinate system. Bringing all palmprint images to a same coordinate system, results in fewer computations during minutia matching. In this paper, using convolutional neural network (CNN) and generalized Hough transform (G…
Bristol Vision Institute（布里斯托尔视觉研究所）；University of Bristol（布里斯托尔大学）
It has recently been demonstrated that spatial resolution adaptation can be integrated within video compression to improve overall coding performance by spatially down-sampling before encoding and super-resolving at the decoder. Significant improvements have been reported when convolutional neural networks (CNNs) were used to perform the resolution up-sampling. However, this approach suffers from high complexity at the decoder due to the employment of CNN-based super-resolution. In this paper, a novel framework is proposed which supports the flexible allocation of complexity between the encode…
This paper presents a predictive model for estimating regularization parameters of diffeomorphic image registration. We introduce a novel framework that automatically determines the parameters controlling the smoothness of diffeomorphic transformations. Our method significantly reduces the effort of parameter tuning, which is time and labor-consuming. To achieve the goal, we develop a predictive model based on deep convolutional neural networks (CNN) that learns the mapping between pairwise images and the regularization parameter of image registration. In contrast to previous methods that esti…
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；Radboud UMC（拉德堡德大学医学中心）；University of Lübeck（吕贝克大学）
While deep learning has achieved significant advances in accuracy for medical image segmentation, its benefits for deformable image registration have so far remained limited to reduced computation times. Previous work has either focused on replacing the iterative optimization of distance and smoothness terms with CNN-layers or using supervised approaches driven by labels. Our method is the first to combine the complementary strengths of global semantic information (represented by segmentation labels) and local distance metrics that help align surrounding structures. We demonstrate significant …
The Hong Kong University of Science and Technology（香港科技大学）
Diffeomorphic deformable image registration is crucial in many medical image studies, as it offers unique, special properties including topology preservation and invertibility of the transformation. Recent deep learning-based deformable image registration methods achieve fast image registration by leveraging a convolutional neural network (CNN) to learn the spatial transformation from the synthetic ground truth or the similarity metric. However, these approaches often ignore the topology preservation of the transformation and the smoothness of the transformation which is enforced by a global s…
Image denoising is always a challenging task in the field of computer vision and image processing. In this paper, we have proposed an encoder-decoder model with direct attention, which is capable of denoising and reconstruct highly corrupted images. Our model consists of an encoder and a decoder, where the encoder is a convolutional neural network and decoder is a multilayer Long Short-Term memory network. In the proposed model, the encoder reads an image and catches the abstraction of that image in a vector, where decoder takes that vector as well as the corrupted image to reconstruct a clean…
UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）
We present a method to predict image deformations based on patch-wise image appearance. Specifically, we design a patch-based deep encoder-decoder network which learns the pixel/voxel-wise mapping between image appearance and registration parameters. Our approach can predict general deformation parameterizations, however, we focus on the large deformation diffeomorphic metric mapping (LDDMM) registration model. By predicting the LDDMM momentum-parameterization we retain the desirable theoretical properties of LDDMM, while reducing computation time by orders of magnitude: combined with patch pr…
University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
Deep convolutional neural networks (CNNs) are state-of-the-art for semantic image segmentation, but typically require many labeled training samples. Obtaining 3D segmentations of medical images for supervised training is difficult and labor intensive. Motivated by classical approaches for joint segmentation and registration we therefore propose a deep learning framework that jointly learns networks for image registration and image segmentation. In contrast to previous work on deep unsupervised image registration, which showed the benefit of weak supervision via image segmentations, our approac…
Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Imperial College London（伦敦帝国理工学院）；University of Pennsylvania（宾夕法尼亚大学）；National University of Singapore（新加坡国立大学）
UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）
We introduce a deep encoder-decoder architecture for image deformation prediction from multimodal images. Specifically, we design an image-patch-based deep network that jointly (i) learns an image similarity measure and (ii) the relationship between image patches and deformation parameters. While our method can be applied to general image registration formulations, we focus on the Large Deformation Diffeomorphic Metric Mapping (LDDMM) registration model. By predicting the initial momentum of the shooting formulation of LDDMM, we preserve its mathematical properties and drastically reduce the c…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Biomedical Research Imaging Center (BRIC)（生物医学研究成像中心）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）
This paper introduces Quicksilver, a fast deformable image registration method. Quicksilver registration for image-pairs works by patch-wise prediction of a deformation model based directly on image appearance. A deep encoder-decoder network is used as the prediction model. While the prediction strategy is general, we focus on predictions for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) model. Specifically, we predict the momentum-parameterization of LDDMM, which facilitates a patch-wise prediction strategy while maintaining the theoretical properties of LDDMM, such as guaranteed…
Image registration is a crucial task in signal processing, but it often encounters issues with stability and efficiency. Non-learning registration approaches rely on optimizing similarity metrics between fixed and moving images, which can be expensive in terms of time and space complexity. This problem can be exacerbated when the images are large or there are significant deformations between them. Recently, deep learning, specifically convolutional neural network (CNN)-based methods, have been explored as an effective solution to the weaknesses of non-learning approaches. To further advance le…
Institute of Mechatronic Systems, Leibniz Universität Hannover（汉诺威莱布尼茨大学机电系统研究所）
We present deformable unsupervised medical image registration using a randomly-initialized deep convolutional neural network (CNN) as regularization prior. Conventional registration methods predict a transformation by minimizing dissimilarities between an image pair. The minimization is usually regularized with manually engineered priors, which limits the potential of the registration. By learning transformation priors from a large dataset, CNNs have achieved great success in deformable registration. However, learned methods are restricted to domain-specific data and the required amounts of me…
Domain-Shift Immunity in Deep Deformable Registration via Local Feature Representations
Kahlert School of Computing（卡尔伯特计算学校）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；University of Utah（犹他大学）；Department of Biomedical Engineering（生物医学工程系）
On the Degrees of Freedom of Gridded Control Points in Learning-Based Medical Image Registration
UCL Hawkes Institute（UCL霍克斯研究所；医学物理与生物医学工程系，伦敦大学学院）；Department of Medical Physics and Biomedical Engineering, University College London（医学成像中心，医学系，伦敦大学学院）；Centre for Medical Imaging, Division of Medicine, University College London（外科与介入科学系，伦敦大学学院）；Division of Surgery and Interventional Science, University College London（癌症研究所，泌尿科，UCL医院，伦敦大学学院）；Cancer Institute, Urology Department, UCL Hospital, University College London（放射科，帝国理工医疗）；Radiology Department, Imperial College Healthcare
Fourier-Net: Fast Image Registration with Band-limited Deformation
School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）；University of Melbourne（墨尔本大学）；University of Cambridge（剑桥大学）；University of Warwick（华威大学）；Institute of Information Computer Engineering, Northeast Forestry University（东北林业大学信息与计算机工程学院）；Alan Turing Institute（阿兰·图灵研究所）
University of Alberta（阿尔伯塔大学）；Servier Virtual Cardiac Centre（施维雅虚拟心脏中心）；Mazankowski Alberta Heart Institute（马赞科夫斯基阿尔伯塔心脏研究所）
Diffeomorphic deformable image registration is one of the crucial tasks in medical image analysis, which aims to find a unique transformation while preserving the topology and invertibility of the transformation. Deep convolutional neural networks (CNNs) have yielded well-suited approaches for image registration by learning the transformation priors from a large dataset. The improvement in the performance of these methods is related to their ability to learn information from several sample medical images that are difficult to obtain and bias the framework to the specific domain of data. In thi…
Indian Institute of Technology Delhi（德里印度理工学院）；Chennai Mathematical Institute（钦奈数学研究所）；Medanta Hospital（梅丹塔医院）
Image registration is a process of aligning two or more images of same objects using geometric transformation. Most of the existing approaches work on the assumption of location invariance. These approaches require object-centric images to perform matching. Further, in absence of intensity level symmetry between the corresponding points in two images, the learning based registration approaches rely on synthetic deformations, which often fail in real scenarios. To address these issues, a combination of convolutional neural networks (CNNs) to perform the desired registration is developed in this…
Multi-Expert Learning Framework with the State Space Model for Optical and SAR Image Registration
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education of China（教育部智能感知与图像理解重点实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州学院）；School of Computer Science, Xidian University（西安电子科技大学计算机学院）；Department of Automation, Tsinghua University（清华大学自动化系）
Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection
School of Electronics and Information Engineering, Wuxi University（无锡大学电子与信息工程学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）
Shanghai Jiao Tong University（上海交通大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Calgary（卡尔加里大学）
The fusion of images taken by heterogeneous sensors helps to enrich the information and improve the quality of imaging. In this article, we present a hybrid model consisting of a convolutional encoder and a Transformer-based decoder to fuse multimodal images. In the encoder, a non-local cross-modal attention block is proposed to capture both local and global dependencies of multiple source images. A branch fusion module is designed to adaptively fuse the features of the two branches. We embed a Transformer module with linear complexity in the decoder to enhance the reconstruction capability of…
A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy
Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA（梅奥诊所放射肿瘤科）；School of Biomedical Engineering, ShanghaiTech University, Shanghai, China（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd., Shanghai 200230, China（上海联合影像智能科技有限公司）；Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China（上海人工智能实验室）；Shanghai Clinical Research and Trial Center, Shanghai 201210, China（上海临床研究与试验中心）
Purpose: The purpose of this paper is to present a method for real-time 2D-3D non-rigid registration using a single fluoroscopic image. Such a method can find applications in surgery, interventional radiology and radiotherapy. By estimating a three-dimensional displacement field from a 2D X-ray image, anatomical structures segmented in the preoperative scan can be projected onto the 2D image, thus providing a mixed reality view. Methods: A dataset composed of displacement fields and 2D projections of the anatomy is generated from the preoperative scan. From this dataset, a neural network is tr…
With the advancement of remote-sensed imaging large volumes of very high resolution land cover images can now be obtained. Automation of object recognition in these 2D images, however, is still a key issue. High intra-class variance and low inter-class variance in Very High Resolution (VHR) images hamper the accuracy of prediction in object recognition tasks. Most successful techniques in various computer vision tasks recently are based on deep supervised learning. In this work, a deep Convolutional Neural Network (CNN) based on symmetric encoder-decoder architecture with skip connections is e…
Institute of Parallel and Distributed Systems, University of Stuttgart（斯图加特大学并行与分布式系统研究所）
Deformable image registration is a fundamental task in medical imaging. Due to the large computational complexity of deformable registration of volumetric images, conventional iterative methods usually face the tradeoff between the registration accuracy and the computation time in practice. In order to boost the registration performance in both accuracy and runtime, we propose a fast convolutional neural network. Specially, to efficiently utilize the memory resources and enlarge the model capacity, we adopt additive forwarding instead of channel concatenation and deepen the network in each enc…
Precise determination of target is an essential procedure in prostate interventions, such as the prostate biopsy, lesion detection and targeted therapy. However, the prostate delineation may be tough in some cases due to tissue ambiguity or lack of partial anatomical boundary. To address this problem, we proposed a weakly supervised registration-based framework for the precise prostate segmentation, by combining convolutional neural network (CNN) with statistical shape model (SSM). To obtain the prostate region, an inception-based neural network (SSM-Net) was firstly exploited to predict the m…
National Research Council of Italy（意大利国家研究委员会）；University Paris-Est Creteil（巴黎东部克雷泰伊大学）
In the last three years, the world has been facing a global crisis caused by Covid-19 pandemic. Medical imaging has been playing a crucial role in the fighting against this disease and saving the human lives. Indeed, CT-scans has proved their efficiency in diagnosing, detecting, and following-up the Covid-19 infection. In this paper, we propose a new Transformer-CNN based approach for Covid-19 infection segmentation from the CT slices. The proposed D-TrAttUnet architecture has an Encoder-Decoder structure, where compound Transformer-CNN encoder and Dual-Decoders are proposed. The Transformer-C…
