Deformable registration is a fundamental task in medical image processing, aiming to achieve precise alignment by establishing nonlinear correspondences between images. Traditional methods offer good adaptability and interpretability but are limited by computational efficiency. Although deep learning approaches have significantly improved registration speed and accuracy, they often lack flexibility and generalizability across different datasets and tasks. In recent years, foundation models have emerged as a promising direction, leveraging large and diverse datasets to learn universal features and transformation patterns for image registration, thus demonstrating strong cross-task transferability. However, these models still face challenges in generalization and robustness when encountering novel anatomical structures, varying imaging conditions, or unseen modalities. To address these limitations, this paper incorporates Sharpness-Aware Minimization (SAM) into foundation models to enhance their generalization and robustness in medical image registration. By optimizing the flatness of the loss landscape, SAM improves model stability across diverse data distributions and strengthens its ability to handle complex clinical scenarios. Experimental results show that foundation models integrated with SAM achieve significant improvements in cross-dataset registration performance, offering new insights for the advancement of medical image registration technology. Our code is available at https://github.com/Promise13/fm_sam}{https://github.com/Promise13/fm\_sam.
Medical Image Registration and Its Application in Retinal Images: A Review
Southern University of Science and Technology(南方科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Singapore Eye Research Institute(新加坡眼科研究所)
Johns Hopkins Medical Institutes(约翰·霍普金斯医学院);Johns Hopkins University(约翰·霍普金斯大学);NVIDIA Corporation(英伟达公司);Duke University Medical Center(杜克大学医学中心);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehensi…
Medical image registration is a difficult problem. Not only a registration algorithm needs to capture both large and small scale image deformations, it also has to deal with global and local image intensity variations. In this paper we describe a new multiresolution elastic image registration method that challenges these difficulties in image registration. To capture large and small scale image deformations, we use both global and local affine transformation algorithms. To address global and local image intensity variations, we apply an image intensity standardization algorithm to correct imag…
We present a fast learning-based algorithm for deformable, pairwise 3D medical image registration. Current registration methods optimize an objective function independently for each pair of images, which can be time-consuming for large data. We define registration as a parametric function, and optimize its parameters given a set of images from a collection of interest. Given a new pair of scans, we can quickly compute a registration field by directly evaluating the function using the learned parameters. We model this function using a convolutional neural network (CNN), and use a spatial transf…
Uncertainty Estimation for Pretrained Medical Image Registration Models via Transformation Equivariance
Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);The Hawkes Institute, University College London(霍克斯研究所,伦敦大学学院);AI Laboratory, Massachusetts Institute of Technology(人工智能实验室,麻省理工学院)
Conventional approaches to image registration consist of time consuming iterative methods. Most current deep learning (DL) based registration methods extract deep features to use in an iterative setting. We propose an end-to-end DL method for registering multimodal images. Our approach uses generative adversarial networks (GANs) that eliminates the need for time consuming iterative methods, and directly generates the registered image with the deformation field. Appropriate constraints in the GAN cost function produce accurately registered images in less than a second. Experiments demonstrate t…
Implicit Deformable Medical Image Registration with Learnable Kernels
Department of Computer Science Interactive Graphics and Simulation Group (IGS), University of Innsbruck, Technikerstraße 21 Innsbruck, Austria(因计算机科学系交互式图形与模拟组(IGS),因斯布鲁克大学);Interventional Oncology-Microinvasive Therapy (SIP), Department of Radiology, Medical University Innsbruck, Innsbruck, Austria(介入肿瘤学-微侵袭疗法(SIP),放射学系,因斯布鲁克医学院)
We propose a registration algorithm for 2D CT/MRI medical images with a new unsupervised end-to-end strategy using convolutional neural networks. The contributions of our algorithm are threefold: (1) We transplant traditional image registration algorithms to an end-to-end convolutional neural network framework, while maintaining the unsupervised nature of image registration problems. The image-to-image integrated framework can simultaneously learn both image features and transformation matrix for registration. (2) Training with additional data without any label can further improve the registra…
The Hong Kong University of Science and Technology(香港科技大学)
Affine registration is indispensable in a comprehensive medical image registration pipeline. However, only a few studies focus on fast and robust affine registration algorithms. Most of these studies utilize convolutional neural networks (CNNs) to learn joint affine and non-parametric registration, while the standalone performance of the affine subnetwork is less explored. Moreover, existing CNN-based affine registration approaches focus either on the local misalignment or the global orientation and position of the input to predict the affine transformation matrix, which are sensitive to spati…
CentraleSupelec(中央理工-高等电力学院);INRIA(法国国家信息与自动化研究所);Universite Paris-Saclay(巴黎萨克雷大学);Imperial College London(帝国理工学院);TheraPanacea
During the last decades, the research community of medical imaging has witnessed continuous advances in image registration methods, which pushed the limits of the state-of-the-art and enabled the development of novel medical procedures. A particular type of image registration problem, known as slice-to-volume registration, played a fundamental role in areas like image guided surgeries and volumetric image reconstruction. However, to date, and despite the extensive literature available on this topic, no survey has been written to discuss this challenging problem. This paper introduces the first…
LLaMA-Reg: Using LLaMA 2 for Unsupervised Medical Image Registration
Faculty of Information Science Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息科学工程与自动化学院);Taizhou Hospital of Zhejiang Province Affiliated with Wenzhou Medical University(温州医科大学附属浙江省台州医院);Enze Hospital, Taizhou Enze Medical Centre (Group)(台州恩泽医疗中心(集团)恩泽医院);The Second Affiliated Hospital of Dalian Medical University(大连医科大学附属第二医院)
In this work, we propose a self-supervised learning method for affine image registration on 3D medical images. Unlike optimisation-based methods, our affine image registration network (AIRNet) is designed to directly estimate the transformation parameters between two input images without using any metric, which represents the quality of the registration, as the optimising function. But since it is costly to manually identify the transformation parameters between any two images, we leverage the abundance of cheap unlabelled data to generate a synthetic dataset for the training of the model. Add…
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Beihang University(北京航空航天大学);Microsoft Research(微软研究院)
3D medical image registration is of great clinical importance. However, supervised learning methods require a large amount of accurately annotated corresponding control points (or morphing), which are very difficult to obtain. Unsupervised learning methods ease the burden of manual annotation by exploiting unlabeled data without supervision. In this paper, we propose a new unsupervised learning method using convolutional neural networks under an end-to-end framework, Volume Tweening Network (VTN), for 3D medical image registration. We propose three innovative technical components: (1) An end-t…
the University of Sydney(悉尼大学);Royal Prince Alfred Hospital(皇家阿尔弗雷德王子医院);Shanghai Jiao Tong University(上海交通大学)
Deformable image registration is fundamental for many medical image analyses. A key obstacle for accurate image registration lies in image appearance variations such as the variations in texture, intensities, and noise. These variations are readily apparent in medical images, especially in brain images where registration is frequently used. Recently, deep learning-based registration methods (DLRs), using deep neural networks, have shown computational efficiency that is several orders of magnitude faster than traditional optimization-based registration methods (ORs). DLRs rely on a globally opt…
Bangladesh University of Engineering and Technology(孟加拉国工程技术大学);Cincinnati Children’s Hospital Medical Center(辛辛那提儿童医院医疗中心);University of Cincinnati College of Medicine(辛辛那提大学医学院);University of Cincinnati(辛辛那提大学)
Image registration is a critical component in the applications of various medical image analyses. In recent years, there has been a tremendous surge in the development of deep learning (DL)-based medical image registration models. This paper provides a comprehensive review of medical image registration. Firstly, a discussion is provided for supervised registration categories, for example, fully supervised, dual supervised, and weakly supervised registration. Next, similarity-based as well as generative adversarial network (GAN)-based registration are presented as part of unsupervised registrat…
DGIST(大邱庆北科学技术院);Yeungnam University College of Medicine(岭南大学医学院);Yeungnam University Hospital(岭南大学医院)
Non-rigid registration is a necessary but challenging task in medical imaging studies. Recently, unsupervised registration models have shown good performance, but they often require a large-scale training dataset and long training times. Therefore, in real world application where only dozens to hundreds of image pairs are available, existing models cannot be practically used. To address these limitations, we propose a novel unsupervised registration model which is integrated with a gradient-based meta learning framework. In particular, we train a meta learner which finds an initialization poin…
Deformable image registration is widely utilized in medical image analysis, but most proposed methods fail in the situation of complex deformations. In this paper, we pre-sent a cascaded feature warping network to perform the coarse-to-fine registration. To achieve this, a shared-weights encoder network is adopted to generate the feature pyramids for the unaligned images. The feature warping registration module is then used to estimate the deformation field at each level. The coarse-to-fine manner is implemented by cascading the module from the bottom level to the top level. Furthermore, the m…
MambaMorph: a Mamba-based Framework for Medical MR-CT Deformable Registration
Image Processing Center, Beihang University, Beijing, China(北京航空航天大学图像处理中心);Robotics Institute, Beihang University, Beijing, China(北京航空航天大学机器人研究所);Tongji Hospital, Huazhong University of Science and Technology, Wuhan, China(华中科技大学同济医院)
We present a new unsupervised learning algorithm, "FAIM", for 3D medical image registration. With a different architecture than the popular "U-net", the network takes a pair of full image volumes and predicts the displacement fields needed to register source to target. Compared with "U-net" based registration networks such as VoxelMorph, FAIM has fewer trainable parameters but can achieve higher registration accuracy as judged by Dice score on region labels in the Mindboggle-101 dataset. Moreover, with the proposed penalty loss on negative Jacobian determinants, FAIM produces deformations with…
Robust Rigid and Non-Rigid Medical Image Registration Using Learnable Edge Kernels
Department of Mathematics, University of Innsbruck(因斯布鲁克大学数学系);Department of Radiology, Medical University of Innsbruck(因斯布鲁克医学大学放射学系);Neuroimaging Research Core Facility, Medical University of Innsbruck(因斯布鲁克医学大学神经影像研究核心设施)
University of Waterloo(滑铁卢大学);Sunnybrook Research Institute(桑尼布鲁克研究所)
We present Recurrence with Correlation Network (RWCNet), a medical image registration network with multi-scale features and a cost volume layer. We demonstrate that these architectural features improve medical image registration accuracy in two image registration datasets prepared for the MICCAI 2022 Learn2Reg Workshop Challenge. On the large-displacement National Lung Screening Test (NLST) dataset, RWCNet is able to achieve a total registration error (TRE) of 2.11mm between corresponding keypoints without instance fine-tuning. On the OASIS brain MRI dataset, RWCNet is able to achieve an avera…
2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding con…
General Vision Encoder Features as Guidance in Medical Image Registration
Klinikum Rechts der Isar, Technical University of Munich(慕尼黑工业大学右岸医院);School of Computation, Information & Technology, Technical University of Munich(慕尼黑工业大学计算、信息与技术学院);Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich(亥姆霍兹慕尼黑中心生物医学影像机器学习研究所);Munich Center for Machine Learning(慕尼黑机器学习中心);Cancer Research UK Cambridge Institute, University of Cambridge(剑桥大学英国癌症研究剑桥研究所);Department of Oncology, University of Cambridge(剑桥大学肿瘤学系);German Cancer Consortium (DKTK), Partner Site Munich(德国癌症联盟慕尼黑合作站点);Department of Computing, Imperial College London(伦敦帝国理工学院计算机系)
In image registration, many efforts have been devoted to the development of alternatives to the popular normalized mutual information criterion. Concurrently to these efforts, an increasing number of works have demonstrated that substantial gains in registration accuracy can also be achieved by aligning structural representations of images rather than images themselves. Following this research path, we propose a new method for mono- and multimodal image registration based on the alignment of regularized vector fields derived from structural information such as gradient vector flow fields, a te…
uniGradICON: A Foundation Model for Medical Image Registration
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Salzburg(萨尔茨堡大学);University Paris-Est(巴黎东部大学);Brigham and Women’s Hospital(布莱根妇女医院);ÉTS Montréal(蒙特利尔工程学院);Boston University(波士顿大学)
Institute of Mechatronic Systems, Leibniz Universität Hannover(汉诺威莱布尼茨大学机电系统研究所)
We present deformable unsupervised medical image registration using a randomly-initialized deep convolutional neural network (CNN) as regularization prior. Conventional registration methods predict a transformation by minimizing dissimilarities between an image pair. The minimization is usually regularized with manually engineered priors, which limits the potential of the registration. By learning transformation priors from a large dataset, CNNs have achieved great success in deformable registration. However, learned methods are restricted to domain-specific data and the required amounts of me…
Acquisition-to-acquisition signal intensity variations (non-standardness) are inherent in MR images. Standardization is a post processing method for correcting inter-subject intensity variations through transforming all images from the given image gray scale into a standard gray scale wherein similar intensities achieve similar tissue meanings. The lack of a standard image intensity scale in MRI leads to many difficulties in tissue characterizability, image display, and analysis, including image segmentation. This phenomenon has been documented well; however, effects of standardization on medi…
iPEAR: Iterative Pyramid Estimation with Attention and Residuals for Deformable Medical Image Registration
College of Software, Jilin University(吉林大学软件学院);Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly, Nanyang Technological University(老年活跃生活卓越研究中⼼(NTU-UBC), 新加坡国立大学);College of Computer Science(计算机科学学院);Technology, Zhejiang University(技术, 浙江大学);Key Laboratory of Symbolic Computation(符号计算重点实验室);Knowledge Engineering of Ministry of Education, College of Computer Science(教育部知识工程重点实验室, 计算机科学学院);Technology, Jilin University(技术, 吉林大学);School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
Institute of Medical Informatics, Universität zu Lübeck(吕贝克大学医学信息学研究所)
Though, deep learning based medical image registration is currently starting to show promising advances, often, it still fells behind conventional frameworks in terms of registration accuracy. This is especially true for applications where large deformations exist, such as registration of interpatient abdominal MRI or inhale-to-exhale CT lung registration. Most current works use U-Net-like architectures to predict dense displacement fields from the input images in different supervised and unsupervised settings. We believe that the U-Net architecture itself to some level limits the ability to p…
University of Freiburg(弗莱堡大学);German Cancer Consortium (DKTK)(德国癌症联盟(DKTK))
Deformable image registration is a very important field of research in medical imaging. Recently multiple deep learning approaches were published in this area showing promising results. However, drawbacks of deep learning methods are the need for a large amount of training datasets and their inability to register unseen images different from the training datasets. One shot learning comes without the need of large training datasets and has already been proven to be applicable to 3D data. In this work we present a one shot registration approach for periodic motion tracking in 3D and 4D datasets.…
Universität zu Lübeck(吕贝克大学);Institute of Medical Informatics(医学信息学研究所)
As in other areas of medical image analysis, e.g. semantic segmentation, deep learning is currently driving the development of new approaches for image registration. Multi-scale encoder-decoder network architectures achieve state-of-the-art accuracy on tasks such as intra-patient alignment of abdominal CT or brain MRI registration, especially when additional supervision, such as anatomical labels, is available. The success of these methods relies to a large extent on the outstanding ability of deep CNNs to extract descriptive visual features from the input images. In contrast to conventional m…
In this paper a fast triangular mesh based registration method is proposed. Having Template and Reference images as inputs, the template image is triangulated using a content adaptive mesh generation algorithm. Considering the pixel values at mesh nodes, interpolated using spline interpolation method for both of the images, the energy functional needed for image registration is minimized. The minimization process was achieved using a mesh based discretization of the distance measure and regularization term which resulted in a sparse system of linear equations, which due to the smaller size in …
Leiden University Medical Center(莱顿大学医学中心);Centrum Wiskunde & Informatica(荷兰数学与计算机科学研究院)
Reliably and physically accurately transferring information between images through deformable image registration with large anatomical differences is an open challenge in medical image analysis. Most existing methods have two key shortcomings: first, they require extensive up-front parameter tuning to each specific registration problem, and second, they have difficulty capturing large deformations and content mismatches between images. There have however been developments that have laid the foundation for potential solutions to both shortcomings. Towards the first shortcoming, a multi-objectiv…
Pyramid Attention Network for Medical Image Registration
School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院);Medical UltraSound Image Computing (MUSIC) Lab(医学超声图像计算实验室);Smart Medical Imaging, Learning and Engineering (SMILE) Lab(智能医学影像、学习与工程实验室)
Institute for Parallel and Distributed Systems, University of Stuttgart(斯图加特大学并行与分布式系统研究所)
Image registration is a fundamental task for medical imaging. Resampling of the intensity values is required during registration and better spatial resolution with finer and sharper structures can improve the resampling performance and hence the registration accuracy. Super-resolution (SR) is an algorithmic technique targeting at spatial resolution enhancement which can achieve an image resolution beyond the hardware limitation. In this work, we consider SR as a preprocessing technique and present a CNN-based resolution enhancement module (REM) which can be easily plugged into the registration…
Southern Medical University(南方医科大学);Shenzhen University(深圳大学)
Deformation field estimation is an important and challenging issue in many medical image registration applications. In recent years, deep learning technique has become a promising approach for simplifying registration problems, and has been gradually applied to medical image registration. However, most existing deep learning registrations do not consider the problem that when the receptive field cannot cover the corresponding features in the moving image and the fixed image, it cannot output accurate displacement values. In fact, due to the limitation of the receptive field, the 3 x 3 kernel h…
Registration networks have shown great application potentials in medical image analysis. However, supervised training methods have a great demand for large and high-quality labeled datasets, which is time-consuming and sometimes impractical due to data sharing issues. Unsupervised image registration algorithms commonly employ intensity-based similarity measures as loss functions without any manual annotations. These methods estimate the parameterized transformations between pairs of moving and fixed images through the optimization of the network parameters during training. However, these metho…
Leiden University Medical Center(莱顿大学医学中心);Centrum Wiskunde & Informatica(荷兰数学与计算机科学研究院)
Finding a realistic deformation that transforms one image into another, in case large deformations are required, is considered a key challenge in medical image analysis. Having a proper image registration approach to achieve this could unleash a number of applications requiring information to be transferred between images. Clinical adoption is currently hampered by many existing methods requiring extensive configuration effort before each use, or not being able to (realistically) capture large deformations. A recent multi-objective approach that uses the Multi-Objective Real-Valued Gene-pool O…
Southeast University(东南大学);Centre de Recherche en Information Biomédicale Sino-Français (CRIBs)(中法生物医学信息研究中心);Univ Rennes(雷恩大学);Inserm(法国国家健康与医学研究院);LTSI - UMR1099(信号与图像处理实验室)
Deep learning-based medical image registration and segmentation joint models utilize the complementarity (augmentation data or weakly supervised data from registration, region constraints from segmentation) to bring mutual improvement in complex scene and few-shot situation. However, further adoption of the joint models are hindered: 1) the diversity of augmentation data is reduced limiting the further enhancement of segmentation, 2) misaligned regions in weakly supervised data disturb the training process, 3) lack of label-based region constraints in few-shot situation limits the registration…
Southeast University(东南大学);Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing(江苏省医学信息处理联合国际研究实验室);University of Western Ontario(西安大略大学)
An effective backbone network is important to deep learning-based Deformable Medical Image Registration (DMIR), because it extracts and matches the features between two images to discover the mutual correspondence for fine registration. However, the existing deep networks focus on single image situation and are limited in registration task which is performed on paired images. Therefore, we advance a novel backbone network, XMorpher, for the effective corresponding feature representation in DMIR. 1) It proposes a novel full transformer architecture including dual parallel feature extraction net…
Image registration plays an important role in medical image analysis. Conventional optimization based methods provide an accurate estimation due to the iterative process at the cost of expensive computation. Deep learning methods such as learn-to-map are much faster but either iterative or coarse-to-fine approach is required to improve accuracy for handling large motions. In this work, we proposed to learn a registration optimizer via a multi-scale neural ODE model. The inference consists of iterative gradient updates similar to a conventional gradient descent optimizer but in a much faster wa…
SAME++: A Self-supervised Anatomical eMbeddings Enhanced medical image registration framework using stable sampling and regularized transformation
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Alibaba Group(阿里巴巴集团);Johns Hopkins University(约翰斯·霍普金斯大学);The First Affiliated Hospital of Zhejiang University(浙江大学附属第一医院)
Efficient Large-Deformation Medical Image Registration via Recurrent Dynamic Correlation
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(脑启发智能科学与技术研究院,复旦大学);Department of Radiology, Leiden University Medical Center(放射科,莱顿大学医学中心)
Are Vision Foundation Models Ready for Out-of-the-Box Medical Image Registration?
Department of Electrical and Computer Engineering, Duke University(电子工程与计算机科学系,杜克大学);Department of Biostatistics and Bioinformatics, Duke University(生物统计学与生物信息学系,杜克大学);Departments of Biostatistics and Bioinformatics, Radiology, Electrical and Computer Engineering, and Computer Science, Duke University(生物统计学与生物信息学系、放射学、电子工程与计算机科学系,杜克大学)
SpikeReg: Energy-Efficient 3D Deformable Medical Image Registration with Spiking Neural Networks
School of Electrical and Computer Engineering, College of Engineering, University of Tehran(德黑兰大学电气与计算机工程学院);Max Planck Institute for Brain Research(马克斯·普朗克脑科学研究所);School of Computer Engineering, Iran University of Science and Technology (IUST)(伊朗科学技术大学计算机工程学院);Department of Electrical and Computer Engineering, University of Waterloo(滑铁卢大学电气与计算机工程系)
Computer Science and Artificial Intelligence Lab, MIT(麻省理工学院计算机科学与人工智能实验室);Meinig School of Biomedical Engineering, Cornell University(康奈尔大学迈尼格生物医学工程学院);Martinos Center for Biomedical Imaging, MGH, HMS(麻省总医院哈佛医学院马蒂诺生物医学成像中心)
We present VoxelMorph, a fast learning-based framework for deformable, pairwise medical image registration. Traditional registration methods optimize an objective function for each pair of images, which can be time-consuming for large datasets or rich deformation models. In contrast to this approach, and building on recent learning-based methods, we formulate registration as a function that maps an input image pair to a deformation field that aligns these images. We parameterize the function via a convolutional neural network (CNN), and optimize the parameters of the neural network on a set of…
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
Medical image registration and segmentation are critical tasks for several clinical procedures. Manual realisation of those tasks is time-consuming and the quality is highly dependent on the level of expertise of the physician. To mitigate that laborious task, automatic tools have been developed where the majority of solutions are supervised techniques. However, in medical domain, the strong assumption of having a well-representative ground truth is far from being realistic. To overcome this challenge, unsupervised techniques have been investigated. However, they are still limited in performan…