A coarse-to-fine framework for unsupervised multi-contrast MR image deformable registration with dual consistency constraint
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心）；Pengcheng Laboratory（鹏城实验室）；Pazhou Lab（琶洲实验室）；Guizhou Provincial People’s Hospital（贵州省人民医院）
Weijian Huang, Hao Yang, Xinfeng Liu, Cheng Li, Ian Zhang, Rongpin Wang, Hairong Zheng, Shanshan Wang
Multi-contrast magnetic resonance (MR) image registration is useful in the clinic to achieve fast and accurate imaging-based disease diagnosis and treatment planning. Nevertheless, the efficiency and performance of the existing registration algorithms can still be improved. In this paper, we propose a novel unsupervised learning-based framework to achieve accurate and efficient multi-contrast MR image registrations. Specifically, an end-to-end coarse-to-fine network architecture consisting of affine and deformable transformations is designed to improve the robustness and achieve end-to-end registration. Furthermore, a dual consistency constraint and a new prior knowledge-based loss function are developed to enhance the registration performances. The proposed method has been evaluated on a clinical dataset containing 555 cases, and encouraging performances have been achieved. Compared to the commonly utilized registration methods, including VoxelMorph, SyN, and LT-Net, the proposed method achieves better registration performance with a Dice score of 0.8397 in identifying stroke lesions. With regards to the registration speed, our method is about 10 times faster than the most competitive method of SyN (Affine) when testing on a CPU. Moreover, we prove that our method can still perform well on more challenging tasks with lacking scanning information data, showing high robustness for the clinical application.
University of Cambridge（剑桥大学）；Cancer Research UK Cambridge Centre（英国癌症研究中心剑桥中心）；Medical University Vienna（维也纳医科大学）；Cambridge University Hospitals National Health Service (NHS) Foundation Trust（剑桥大学医院国民保健服务基金会信托）；Eindhoven University of Technology（埃因霍温理工大学）；Leiden University（莱顿大学）；University College London（伦敦大学学院）；Cancer Research UK Cambridge Institute（英国癌症研究中心剑桥研究所）
This study investigates the use of the unsupervised deep learning framework VoxelMorph for deformable registration of longitudinal abdominopelvic CT images acquired in patients with bone metastases from breast cancer. The CT images were refined prior to registration by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of VoxelMorph when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images. In a 4-fold cross-validation scheme…
Tsinghua University（清华大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Deformable image registration (DIR) is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in DIR. Most deep learning approaches use the so-called mono-stream "high-to-low, low-to-high" network structure, and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of de…
Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）；NVIDIA Corporation（英伟达公司）
Transformers have recently shown promise for medical image applications, leading to an increasing interest in developing such models for medical image registration. Recent advancements in designing registration Transformers have focused on using cross-attention (CA) to enable a more precise understanding of spatial correspondences between moving and fixed images. Here, we propose a novel CA mechanism that computes windowed attention using deformable windows. In contrast to existing CA mechanisms that require intensive computational complexity by either computing CA globally or locally with a f…
The Hong Kong University of Science and Technology（香港科技大学）
Registration of pre-operative and post-recurrence brain images is often needed to evaluate the effectiveness of brain gliomas treatment. While recent deep learning-based deformable registration methods have achieved remarkable success with healthy brain images, most of them would be unable to accurately align images with pathologies due to the absent correspondences in the reference image. In this paper, we propose a deep learning-based deformable registration method that jointly estimates regions with absent correspondence and bidirectional deformation fields. A forward-backward consistency c…
The Hong Kong University of Science and Technology（香港科技大学）
Recent deep learning-based methods have shown promising results and runtime advantages in deformable image registration. However, analyzing the effects of hyperparameters and searching for optimal regularization parameters prove to be too prohibitive in deep learning-based methods. This is because it involves training a substantial number of separate models with distinct hyperparameter values. In this paper, we propose a conditional image registration method and a new self-supervised learning paradigm for deep deformable image registration. By learning the conditional features that are correla…
Johns Hopkins University（约翰斯·霍普金斯大学）；National Institute on Aging（国家老龄化研究所）；National Institute of Health（美国国立卫生研究院）
The majority of deep learning (DL) based deformable image registration methods use convolutional neural networks (CNNs) to estimate displacement fields from pairs of moving and fixed images. This, however, requires the convolutional kernels in the CNN to not only extract intensity features from the inputs but also understand image coordinate systems. We argue that the latter task is challenging for traditional CNNs, limiting their performance in registration tasks. To tackle this problem, we first introduce Coordinate Translator, a differentiable module that identifies matched features between…
Johns Hopkins University（约翰斯·霍普金斯大学）；PAII Inc.（PAII公司）；Ping An Technology（平安科技）；The First Affiliated Hospital Zhejiang University（浙江大学医学院附属第一医院）
In this work, we introduce a fast and accurate method for unsupervised 3D medical image registration. This work is built on top of a recent algorithm SAM, which is capable of computing dense anatomical/semantic correspondences between two images at the pixel level. Our method is named SAME, which breaks down image registration into three steps: affine transformation, coarse deformation, and deep deformable registration. Using SAM embeddings, we enhance these steps by finding more coherent correspondences, and providing features and a loss function with better semantic guidance. We collect a mu…
Unsupervised Deformable Image Registration with Local-Global Attention and Image Decomposition
Institute of Medical Technology, Peking University Health Science Center（北京大学医学部医学技术研究所）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）；Department of Radiology, Peking University Third Hospital（北京大学第三医院放射科）；Department of Pediatrics, Peking University First Hospital（北京大学第一医院儿科）；Pediatric Epilepsy Center, Peking University First Hospital（北京大学第一医院儿童癫痫中心）；School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Deformable image registration is one of the fundamental tasks in medical imaging. Classical registration algorithms usually require a high computational cost for iterative optimizations. Although deep-learning-based methods have been developed for fast image registration, it is still challenging to obtain realistic continuous deformations from a moving image to a fixed image with less topological folding problem. To address this, here we present a novel diffusion-model-based image registration method, called DiffuseMorph. DiffuseMorph not only generates synthetic deformed images through revers…
University of Edinburgh（爱丁堡大学）；Imperial College London（帝国理工学院）
Deep learning-based image registration approaches have shown competitive performance and run-time advantages compared to conventional image registration methods. However, existing learning-based approaches mostly require to train separate models with respect to different regularization hyperparameters for manual hyperparameter searching and often do not allow spatially-variant regularization. In this work, we propose a learning-based registration approach based on a novel conditional spatially adaptive instance normalization (CSAIN) to address these challenges. The proposed method introduces a…
Deformable templates are essential to large-scale medical image registration, segmentation, and population analysis. Current conventional and deep network-based methods for template construction use only regularized registration objectives and often yield templates with blurry and/or anatomically implausible appearance, confounding downstream biomedical interpretation. We reformulate deformable registration and conditional template estimation as an adversarial game wherein we encourage realism in the moved templates with a generative adversarial registration framework conditioned on flexible i…
2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization
ImFusion GmbH（ImFusion有限公司）；Technical University of Munich（慕尼黑工业大学）；Institute of Pathology, Heinrich Heine University and University Hospital of Düsseldorf（海因里希·海涅大学与杜塞尔多夫大学医院病理研究所）；Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut（弗劳恩霍夫高速动力学研究所（恩斯特·马赫研究所））
Huazhong University of Science and Technology（华中科技大学）；College of Life Science and Technology, Huazhong University of Science and Technology（华中科技大学生命科学与技术学院）
Deformable medical image registration plays an important role in clinical diagnosis and treatment. Recently, the deep learning (DL) based image registration methods have been widely investigated and showed excellent performance in computational speed. However, these methods cannot provide enough registration accuracy because of insufficient ability in representing both the global and local features of the moving and fixed images. To address this issue, this paper has proposed the transformer based image registration method. This method uses the distinctive transformer to extract the global and…
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；Radboud UMC（拉德堡德大学医学中心）；University of Lübeck（吕贝克大学）
While deep learning has achieved significant advances in accuracy for medical image segmentation, its benefits for deformable image registration have so far remained limited to reduced computation times. Previous work has either focused on replacing the iterative optimization of distance and smoothness terms with CNN-layers or using supervised approaches driven by labels. Our method is the first to combine the complementary strengths of global semantic information (represented by segmentation labels) and local distance metrics that help align surrounding structures. We demonstrate significant …
Image Sciences Institute, University Medical Center Utrecht and Utrecht University（乌得勒支大学医学中心与乌得勒支大学影像科学研究所）；Division of Image Processing of the Leiden University Medical Center（莱顿大学医学中心图像处理部）
Image registration, the process of aligning two or more images, is the core technique of many (semi-)automatic medical image analysis tasks. Recent studies have shown that deep learning methods, notably convolutional neural networks (ConvNets), can be used for image registration. Thus far training of ConvNets for registration was supervised using predefined example registrations. However, obtaining example registrations is not trivial. To circumvent the need for predefined examples, and thereby to increase convenience of training ConvNets for image registration, we propose the Deep Learning Im…
Multimodal Deformable Image Registration for Long-COVID Analysis Based on Progressive Alignment and Multi-perspective Loss
University of Oxford（牛津大学）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）；Big Data Institute, University of Oxford（牛津大学大数据研究所）；Oxford University Hospitals NHS Foundation Trust（牛津大学医院 NHS 基金会信托基金）；Oxford Radiology Research Unit（牛津放射学研究单元）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Medical Center（峨山医学中心）
Medical image registration is one of the key processing steps for biomedical image analysis such as cancer diagnosis. Recently, deep learning based supervised and unsupervised image registration methods have been extensively studied due to its excellent performance in spite of ultra-fast computational time compared to the classical approaches. In this paper, we present a novel unsupervised medical image registration method that trains deep neural network for deformable registration of 3D volumes using a cycle-consistency. Thanks to the cycle consistency, the proposed deep neural networks can t…
Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心图像科学研究所）；Division of Image Processing, Leiden University Medical Center（莱顿大学医学中心图像处理部）
In this work we propose a deep learning network for deformable image registration (DIRNet). The DIRNet consists of a convolutional neural network (ConvNet) regressor, a spatial transformer, and a resampler. The ConvNet analyzes a pair of fixed and moving images and outputs parameters for the spatial transformer, which generates the displacement vector field that enables the resampler to warp the moving image to the fixed image. The DIRNet is trained end-to-end by unsupervised optimization of a similarity metric between input image pairs. A trained DIRNet can be applied to perform registration …
Non-iterative Coarse-to-fine Transformer Networks for Joint Affine and Deformable Image Registration
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）；Med-X Research Institute, Shanghai Jiao Tong University（上海交通大学Med-X研究院）
University College London（伦敦大学学院）；Massachusetts General Hospital（马萨诸塞总医院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Landmark correspondences are a widely used type of gold standard in image registration. However, the manual placement of corresponding points is subject to high inter-user variability in the chosen annotated locations and in the interpretation of visual ambiguities. In this paper, we introduce a principled strategy for the construction of a gold standard in deformable registration. Our framework: (i) iteratively suggests the most informative location to annotate next, taking into account its redundancy with previous annotations; (ii) extends traditional pointwise annotations by accounting for …
Deformable image registration, i.e., the task of aligning multiple images into one coordinate system by non-linear transformation, serves as an essential preprocessing step for neuroimaging data. Recent research on deformable image registration is mainly focused on improving the registration accuracy using multi-stage alignment methods, where the source image is repeatedly deformed in stages by a same neural network until it is well-aligned with the target image. Conventional methods for multi-stage registration can often blur the source image as the pixel/voxel values are repeatedly interpola…
Ecole Centrale de Nantes（南特中央理工学院）；Keosys Medical Imaging（Keosys医学影像公司）；University of Nantes（南特大学）；University of Angers（昂热大学）；ICO Cancer Center（ICO癌症中心）
Longitudinal image registration is challenging and has not yet benefited from major performance improvements thanks to deep-learning. Inspired by Deep Image Prior, this paper introduces a different use of deep architectures as regularizers to tackle the image registration question. We propose a subject-specific deformable registration method called MIRRBA, relying on a deep pyramidal architecture to be the prior parametric model constraining the deformation field. Diverging from the supervised learning paradigm, MIRRBA does not require a learning database, but only the pair of images to be reg…
University of North Carolina（北卡罗来纳大学）；Shanghai Jiao Tong University（上海交通大学）
Deformable image registration is fundamental to longitudinal and population analysis. Geometric alignment of the infant brain MR images is challenging, owing to rapid changes in image appearance in association with brain development. In this paper, we propose an infant-dedicated deep registration network that uses the auto-context strategy to gradually refine the deformation fields to obtain highly accurate correspondences. Instead of training multiple registration networks, our method estimates the deformation fields by invoking a single network multiple times for iterative deformation refine…
Robust deformable image registration using cycle-consistent implicit representations
Amsterdam UMC location University of Amsterdam（阿姆斯特丹大学医学中心（阿姆斯特丹大学））；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Amsterdam Gastroenterology Endocrinology Metabolism（阿姆斯特丹胃肠病学与内分泌代谢研究所）；Cancer Center Amsterdam（阿姆斯特丹癌症中心）
University of Bern（伯尔尼大学）；CentraleSupélec（中央理工-高等电力学院）；Université Paris-Saclay（巴黎-萨克雷大学）；Groupe Hospitalier Cochin-Hôtel Dieu（科钦-主宫医院集团）；Université Paris Descartes（巴黎笛卡尔大学）；TheraPanacea
Image registration and in particular deformable registration methods are pillars of medical imaging. Inspired by the recent advances in deep learning, we propose in this paper, a novel convolutional neural network architecture that couples linear and deformable registration within a unified architecture endowed with near real-time performance. Our framework is modular with respect to the global transformation component, as well as with respect to the similarity function while it guarantees smooth displacement fields. We evaluate the performance of our network on the challenging problem of MRI …
Learning a Model-Driven Variational Network for Deformable Image Registration
University of Birmingham（伯明翰大学）；Alan Turing Institute（阿兰·图灵研究所）；Shenzhen University（深圳大学）；Imperial College London（伦敦帝国学院）；MRC London Institute of Medical Sciences（英国医学研究理事会伦敦医学科学研究所）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；The Catholic University of Korea（韩国加图立大学）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Medical Center（峨山医疗中心）；Korea Automotive Technology Institute（韩国汽车技术研究院）
Image registration is a fundamental task in medical image analysis. Recently, deep learning based image registration methods have been extensively investigated due to their excellent performance despite the ultra-fast computational time. However, the existing deep learning methods still have limitation in the preservation of original topology during the deformation with registration vector fields. To address this issues, here we present a cycle-consistent deformable image registration. The cycle consistency enhances image registration performance by providing an implicit regularization to pres…
Domain-Shift Immunity in Deep Deformable Registration via Local Feature Representations
Kahlert School of Computing（卡尔伯特计算学校）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；University of Utah（犹他大学）；Department of Biomedical Engineering（生物医学工程系）
Fraunhofer Institute for Medical Image Computing (MEVIS)（弗劳恩霍夫医学图像计算研究所（MEVIS））；Universität zu Lübeck（吕贝克大学）
We present a highly parallel method for accurate and efficient variational deformable 3D image registration on a consumer-grade graphics processing unit (GPU). We build on recent matrix-free variational approaches and specialize the concepts to the massively-parallel manycore architecture provided by the GPU. Compared to a parallel and optimized CPU implementation, this allows us to achieve an average speedup of 32.53 on 986 real-world CT thorax-abdomen follow-up scans. At a resolution of approximately $256^3$ voxels, the average runtime is 1.99 seconds for the full registration. On the public…
AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）
Robarts Research Institute（罗巴茨研究所）；Western University（韦仕敦大学）
Optimization techniques have been widely used in deformable registration, allowing for the incorporation of similarity metrics with regularization mechanisms. These regularization mechanisms are designed to mitigate the effects of trivial solutions to ill-posed registration problems and to otherwise ensure the resulting deformation fields are well-behaved. This paper introduces a novel deformable registration algorithm, RANCOR, which uses iterative convexification to address deformable registration problems under total-variation regularization. Initial comparative results against four state-of…
A Hybrid Deep Feature-Based Deformable Image Registration Method for Pathology Images
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Mathematics, Sun Yat-sen University（中山大学数学学院）；Stanford University（斯坦福大学）；Guangdong Medical University（广东医科大学）；Xiaohe Healthcare ByteDance（字节跳动小河医疗）
Ryerson University（瑞尔森大学）；University of Toronto（多伦多大学）；Institute for Biomedical Engineering, Science and Technology (iBEST)（生物医学工程、科学与技术研究所（iBEST））
We propose FlowReg, a deep learning-based framework for unsupervised image registration for neuroimaging applications. The system is composed of two architectures that are trained sequentially: FlowReg-A which affinely corrects for gross differences between moving and fixed volumes in 3D followed by FlowReg-O which performs pixel-wise deformations on a slice-by-slice basis for fine tuning in 2D. The affine network regresses the 3D affine matrix based on a correlation loss function that enforces global similarity. The deformable network operates on 2D image slices based on the optical flow netw…
University of Lübeck（吕贝克大学）；Fraunhofer MEVIS（弗劳恩霍夫医学影像计算研究所）
Low-rank and sparse decompositions and robust PCA (RPCA) are highly successful techniques in image processing and have recently found use in groupwise image registration. In this paper, we investigate the drawbacks of the most common RPCA-dissimi\-larity metric in image registration and derive an improved version. In particular, this new metric models low-rank requirements through explicit constraints instead of penalties and thus avoids the pitfalls of the established metric. Equipped with total variation regularization, we present a theoretically justified multilevel scheme based on first-or…
Deformable image registration is a fundamental task in medical image analysis, aiming to establish a dense and non-linear correspondence between a pair of images. Previous deep-learning studies usually employ supervised neural networks to directly learn the spatial transformation from one image to another, requiring task-specific ground-truth registration for model training. Due to the difficulty in collecting precise ground-truth registration, implementation of these supervised methods is practically challenging. Although several unsupervised networks have been recently developed, these metho…
We present INSPIRE, a top-performing general-purpose method for deformable image registration. INSPIRE brings distance measures which combine intensity and spatial information into an elastic B-splines-based transformation model and incorporates an inverse inconsistency penalization supporting symmetric registration performance. We introduce several theoretical and algorithmic solutions which provide high computational efficiency and thereby applicability of the proposed framework in a wide range of real scenarios. We show that INSPIRE delivers highly accurate, as well as stable and robust reg…
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
Deformable MRI Sequence Registration for AI-based Prostate Cancer Diagnosis
Radboudumc（拉德堡德大学医学中心）；Fraunhofer MEVIS（弗劳恩霍夫梅维斯研究所）；Universität zu Lübeck（吕贝克大学）；University Medical Center Groningen（格罗宁根大学医学中心）；Netherlands Cancer Institute（荷兰癌症研究所）；Norwegian University of Science and Technology（挪威科技大学）
Cluster of Excellence – ML for Science（卓越集群——科学机器学习）；University of Tübingen（蒂宾根大学）；Faculty of Health Sciences and Medicine, University of Lucerne（卢塞恩大学健康科学与医学院）；University of Lübeck（吕贝克大学）
New York University（纽约大学）；Hyperfine Research（Hyperfine研究公司）；Yale University（耶鲁大学）
Establishing voxelwise semantic correspondence across distinct imaging modalities is a foundational yet formidable computer vision task. Current multi-modality registration techniques maximize hand-crafted inter-domain similarity functions, are limited in modeling nonlinear intensity-relationships and deformations, and may require significant re-engineering or underperform on new tasks, datasets, and domain pairs. This work presents ContraReg, an unsupervised contrastive representation learning approach to multi-modality deformable registration. By projecting learned multi-scale local patch fe…
University of Oxford（牛津大学）；Institute of Biomedical Engineering（生物医学工程研究所）
Recent successes in deep learning based deformable image registration (DIR) methods have demonstrated that complex deformation can be learnt directly from data while reducing computation time when compared to traditional methods. However, the reliance on fully linear convolutional layers imposes a uniform sampling of pixel/voxel locations which ultimately limits their performance. To address this problem, we propose a novel approach of learning a continuous warp of the source image. Here, the required deformation vector fields are obtained from a concatenated linear and non-linear convolution …
Research institute for signals, systems and computational intelligence, sinc(i), FICH-UNL/CONICET（信号、系统与计算智能研究所 sinc(i)，FICH-UNL/CONICET）；University of Oxford（牛津大学）；TheraPanacea；CentraleSupelec（中央理工-高等电力学院）；Universite Paris-Saclay（巴黎-萨克雷大学）
Deformable registration has been one of the pillars of biomedical image computing. Conventional approaches refer to the definition of a similarity criterion that, once endowed with a deformation model and a smoothness constraint, determines the optimal transformation to align two given images. The definition of this metric function is among the most critical aspects of the registration process. We argue that incorporating semantic information (in the form of anatomical segmentation maps) into the registration process will further improve the accuracy of the results. In this paper, we propose a…
Deformable image registration is able to achieve fast and accurate alignment between a pair of images and thus plays an important role in many medical image studies. The current deep learning (DL)-based image registration approaches directly learn the spatial transformation from one image to another by leveraging a convolutional neural network, requiring ground truth or similarity metric. Nevertheless, these methods only use a global similarity energy function to evaluate the similarity of a pair of images, which ignores the similarity of regions of interest (ROIs) within images. Moreover, DL-…
