Recent deep learning-based methods for medical image registration achieve results that are competitive with conventional optimization algorithms at reduced run times. However, deep neural networks generally require plenty of labeled training data and are vulnerable to domain shifts between training and test data. While typical intensity shifts can be mitigated by keypoint-based registration, these methods still suffer from geometric domain shifts, for instance, due to different fields of view. As a remedy, in this work, we present a novel approach to geometric domain adaptation for image registration, adapting a model from a labeled source to an unlabeled target domain. We build on a keypoint-based registration model, combining graph convolutions for geometric feature learning with loopy belief optimization, and propose to reduce the domain shift through self-ensembling. To this end, we embed the model into the Mean Teacher paradigm. We extend the Mean Teacher to this context by 1) adapting the stochastic augmentation scheme and 2) combining learned feature extraction with differentiable optimization. This enables us to guide the learning process in the unlabeled target domain by enforcing consistent predictions of the learning student and the temporally averaged teacher model. We evaluate the method for exhale-to-inhale lung CT registration under two challenging adaptation scenarios (DIR-Lab 4D CT to COPD, COPD to Learn2Reg). Our method consistently improves on the baseline model by 50%/47% while even matching the accuracy of models trained on target data. Source code is available at https://github.com/multimodallearning/registration-da-mean-teacher.
Deformable registration continues to be one of the key challenges in medical image analysis. While iconic registration methods have started to benefit from the recent advances in medical deep learning, the same does not yet apply for the registration of point sets, e.g. registration based on surfaces, keypoints or landmarks. This is mainly due to the restriction of the convolution operator in modern CNNs to densely gridded input. However, with the newly developed methods from the field of geometric deep learning suitable tools are now emerging, which enable powerful analysis of medical data on…
BrainMorph: A Foundational Keypoint Model for Robust and Flexible Brain MRI Registration
Department of Radiology, Weill Cornell Medicine, New York City, NY, USA（韦尔·科恩医学中心放射科）；School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA（康奈尔大学电气与计算机工程学院）；Computer Science and Artificial Intelligence Lab at the Massachusetts Institute of Technology, Cambridge, MA, USA（麻省理工学院计算机科学与人工智能实验室）
本文提出基于 KeyMorph 的 BrainMorph 关键点基础模型，利用超十万个 3D 脑 MRI 体积训练，实现鲁棒、可解释、快速可控的多模态成对与组级刚性、仿射及非线性配准。
Computational anatomy allows the quantitative analysis of organs in medical images. However, most analysis is constrained to the millimeter scale because of the limited resolution of clinical computed tomography (CT). X-ray microtomography ($μ$CT) on the other hand allows imaging of ex-vivo tissues at a resolution of tens of microns. In this work, we use clinical CT to image lung cancer patients before partial pneumonectomy (resection of pathological lung tissue). The resected specimen is prepared for $μ$CT imaging at a voxel resolution of 50 $μ$m (0.05 mm). This high-resolution image of the l…
Quantitative analysis of fetal lung Diffusion-Weighted MRI (DWI) data shows potential in providing quantitative imaging biomarkers that indirectly reflect fetal lung maturation. However, fetal motion during the acquisition hampered quantitative analysis of the acquired DWI data and, consequently, reliable clinical utilization. We introduce qDWI-morph, an unsupervised deep-neural-network architecture for motion compensated quantitative DWI (qDWI) analysis. Our approach couples a registration sub-network with a quantitative DWI model fitting sub-network. We simultaneously estimate the qDWI param…
To deform or not: treatment-aware longitudinal registration for breast DCE-MRI during neoadjuvant chemotherapy via unsupervised keypoints detection
Radboud University Medical Centre（拉德堡德大学医学中心）；The Netherlands Cancer Institute（荷兰癌症研究所）；Macao Polytechnic University（澳门理工大学）；Maastricht University Medical Centre（马斯特里赫特大学医学中心）
Columbia University（哥伦比亚大学）；Columbia University Medical Center（哥伦比亚大学医学中心）；University of Iowa（爱荷华大学）
Unsupervised learning-based medical image registration approaches have witnessed rapid development in recent years. We propose to revisit a commonly ignored while simple and well-established principle: recursive refinement of deformation vector fields across scales. We introduce a recursive refinement network (RRN) for unsupervised medical image registration, to extract multi-scale features, construct normalized local cost correlation volume and recursively refine volumetric deformation vector fields. RRN achieves state of the art performance for 3D registration of expiratory-inspiratory pairs…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Kitware, Inc.（Kitware公司）
Registration is widely used in image-guided therapy and image-guided surgery to estimate spatial correspondences between organs of interest between planning and treatment images. However, while high-quality computed tomography (CT) images are often available at planning time, limited angle acquisitions are frequently used during treatment because of radiation concerns or imaging time constraints. This requires algorithms to register CT images based on limited angle acquisitions. We, therefore, formulate a 3D/2D registration approach which infers a 3D deformation based on measured projections a…
Deep learning and classical computer vision techniques in medical image analysis: Case studies on brain MRI tissue segmentation, lung CT COPD registration, and skin lesion classification
University of Exeter（埃克塞特大学）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Sorbonne Université（索邦大学）
Institute of Medical Informatics, Universität zu Lübeck（吕贝克大学医学信息学研究所）
Though, deep learning based medical image registration is currently starting to show promising advances, often, it still fells behind conventional frameworks in terms of registration accuracy. This is especially true for applications where large deformations exist, such as registration of interpatient abdominal MRI or inhale-to-exhale CT lung registration. Most current works use U-Net-like architectures to predict dense displacement fields from the input images in different supervised and unsupervised settings. We believe that the U-Net architecture itself to some level limits the ability to p…
Renmin University of China（中国人民大学）；Visionary Intelligence Ltd（远见智能有限公司）；Tongren Hospital（同仁医院）
For retinal image matching (RIM), we propose SuperRetina, the first end-to-end method with jointly trainable keypoint detector and descriptor. SuperRetina is trained in a novel semi-supervised manner. A small set of (nearly 100) images are incompletely labeled and used to supervise the network to detect keypoints on the vascular tree. To attack the incompleteness of manual labeling, we propose Progressive Keypoint Expansion to enrich the keypoint labels at each training epoch. By utilizing a keypoint-based improved triplet loss as its description loss, SuperRetina produces highly discriminativ…
We propose a registration algorithm for 2D CT/MRI medical images with a new unsupervised end-to-end strategy using convolutional neural networks. The contributions of our algorithm are threefold: (1) We transplant traditional image registration algorithms to an end-to-end convolutional neural network framework, while maintaining the unsupervised nature of image registration problems. The image-to-image integrated framework can simultaneously learn both image features and transformation matrix for registration. (2) Training with additional data without any label can further improve the registra…
Purpose: The purpose of this paper is to present a method for real-time 2D-3D non-rigid registration using a single fluoroscopic image. Such a method can find applications in surgery, interventional radiology and radiotherapy. By estimating a three-dimensional displacement field from a 2D X-ray image, anatomical structures segmented in the preoperative scan can be projected onto the 2D image, thus providing a mixed reality view. Methods: A dataset composed of displacement fields and 2D projections of the anatomy is generated from the preoperative scan. From this dataset, a neural network is tr…
GLIDE-Reg: Global-to-Local Deformable Registration Using Co-Optimized Foundation and Handcrafted Features
1 Medical \& Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 2 Magnetic Resonance Research Labs, Department of Radiological Sciences, David Geffen School of Medicine at UCLA, USA 3 Department of Radiation Oncology, David Geffen School of Medicine at UCLA, Los Angeles, USA
Universität zu Lübeck（吕贝克大学）；Institute of Medical Informatics（医学信息学研究所）
As in other areas of medical image analysis, e.g. semantic segmentation, deep learning is currently driving the development of new approaches for image registration. Multi-scale encoder-decoder network architectures achieve state-of-the-art accuracy on tasks such as intra-patient alignment of abdominal CT or brain MRI registration, especially when additional supervision, such as anatomical labels, is available. The success of these methods relies to a large extent on the outstanding ability of deep CNNs to extract descriptive visual features from the input images. In contrast to conventional m…
Accurate delineation of pathological lungs from computed tomography (CT) images remains mostly unsolved because available methods fail to provide a reliable generic solution due to high variability of abnormality appearance. Local descriptor-based classification methods have shown to work well in annotating pathologies; however, these methods are usually computationally intensive which restricts their widespread use in real-time or near-real-time clinical applications. In this paper, we present a novel approach for fast, accurate, reliable segmentation of pathological lungs from CT scans by co…
Unsupervised training of keypoint-agnostic descriptors for flexible retinal image registration
Grupo VARPA, Instituto de Investigacion Biomédica de A Coruña (INIBIC), Universidade da Coruña（VARPA组，阿维拉生物医学研究所（INIBIC），科鲁纳大学）；Departamento de Ciencias de la Computación y Tecnologías de la Información, Universidade da Coruña（计算机科学与信息技术部门，科鲁纳大学）
Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所MEVIS）；Radboud University Medical Center（拉德堡德大学医学中心）
Deep-learning-based registration methods emerged as a fast alternative to conventional registration methods. However, these methods often still cannot achieve the same performance as conventional registration methods because they are either limited to small deformation or they fail to handle a superposition of large and small deformations without producing implausible deformation fields with foldings inside. In this paper, we identify important strategies of conventional registration methods for lung registration and successfully developed the deep-learning counterpart. We employ a Gaussian-py…
In ophthalmological imaging, multiple imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography, are often involved to make a diagnosis of retinal disease. Multi-modal retinal registration techniques can assist ophthalmologists by providing a pixel-based comparison of aligned vessel structures in images from different modalities or acquisition times. To this end, we propose an end-to-end trainable deep learning method for multi-modal retinal image registration. Our method extracts convolutional features from the vessel struc…
Data-Driven Tissue- and Subject-Specific Elastic Regularization for Medical Image Registration
School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich（亥姆霍兹慕尼黑生物医学成像机器学习研究所）；German Cancer Consortium (DKTK)（德国癌症联盟）；School of Biomedical Engineering and Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与成像科学学院）
A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints
Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）；Weill Cornell Medical School（威尔康奈尔医学院）；Iterative Scopes（Iterative Scopes公司）；Massachusetts Institute of Technology（麻省理工学院）；Massachusetts General Hospital（麻省总医院）
University of Waterloo（滑铁卢大学）；Sunnybrook Research Institute（桑尼布鲁克研究所）
We present Recurrence with Correlation Network (RWCNet), a medical image registration network with multi-scale features and a cost volume layer. We demonstrate that these architectural features improve medical image registration accuracy in two image registration datasets prepared for the MICCAI 2022 Learn2Reg Workshop Challenge. On the large-displacement National Lung Screening Test (NLST) dataset, RWCNet is able to achieve a total registration error (TRE) of 2.11mm between corresponding keypoints without instance fine-tuning. On the OASIS brain MRI dataset, RWCNet is able to achieve an avera…
A Novel Streamline-based diffusion MRI Tractography Registration Method with Probabilistic Keypoint Detection
University of Electronic Science and Technology of China（电子科技大学）；Nanjing University of Science and Technology（南京理工大学）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所（MEVIS））；University of Lübeck（吕贝克大学）；Anthropics Technology Ltd.（Anthropics科技有限公司）
We present a novel algorithm for the registration of pulmonary CT scans. Our method is designed for large respiratory motion by integrating sparse keypoint correspondences into a dense continuous optimization framework. The detection of keypoint correspondences enables robustness against large deformations by jointly optimizing over a large number of potential discrete displacements, whereas the dense continuous registration achieves subvoxel alignment with smooth transformations. Both steps are driven by the same normalized gradient fields data term. We employ curvature regularization and a v…
A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration
Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）；Inria（法国国家信息与自动化研究所）；Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（巴黎大脑研究所 - ICM, 法国国家科学研究中心, 法国国家健康与医学研究院, 巴黎公立医院集团, 皮提耶-萨勒佩特里医院）；Technical University of Munich（慕尼黑工业大学）
Tongji University（同济大学）；Technische Universität München（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）；Westwell lab（西井实验室）
Point cloud registration is a fundamental problem in 3D computer vision. Outdoor LiDAR point clouds are typically large-scale and complexly distributed, which makes the registration challenging. In this paper, we propose an efficient hierarchical network named HRegNet for large-scale outdoor LiDAR point cloud registration. Instead of using all points in the point clouds, HRegNet performs registration on hierarchically extracted keypoints and descriptors. The overall framework combines the reliable features in deeper layer and the precise position information in shallower layers to achieve robu…
University of Pennsylvania（宾夕法尼亚大学）；University of Minnesota（明尼苏达大学）
This paper presents a semi-supervised learning framework to train a keypoint detector using multiview image streams given the limited labeled data (typically $<$4\%). We leverage the complementary relationship between multiview geometry and visual tracking to provide three types of supervisionary signals to utilize the unlabeled data: (1) keypoint detection in one view can be supervised by other views via the epipolar geometry; (2) a keypoint moves smoothly over time where its optical flow can be used to temporally supervise consecutive image frames to each other; (3) visible keypoint in one v…
We tackle here the problem of multimodal image non-rigid registration, which is of prime importance in remote sensing and medical imaging. The difficulties encountered by classical registration approaches include feature design and slow optimization by gradient descent. By analyzing these methods, we note the significance of the notion of scale. We design easy-to-train, fully-convolutional neural networks able to learn scale-specific features. Once chained appropriately, they perform global registration in linear time, getting rid of gradient descent schemes by predicting directly the deformat…
Université de Lyon（里昂大学）；INSA Lyon（里昂国立应用科学学院）；Université Claude Bernard Lyon 1（里昂第一大学）；UJM-Saint Etienne（圣艾蒂安让·莫内大学）；CNRS（法国国家科学研究中心）；Inserm（法国国家健康与医学研究院）；Hospices Civils de Lyon（里昂民用医院集团）
We present a novel algorithm for Fast Registration Of image Groups (FROG), applied to large 3D image groups. Our approach extracts 3D SURF keypoints from images, computes matched pairs of keypoints and registers the group by minimizing pair distances in a hubless way i.e. without computing any central mean image. Using keypoints significantly reduces the problem complexity compared to voxel-based approaches, and enables us to provide an in-core global optimization, similar to the Bundle Adjustment for 3D reconstruction. As we aim to register images of different patients, the matching step yiel…
The majority of current research in deep learning based image registration addresses inter-patient brain registration with moderate deformation magnitudes. The recent Learn2Reg medical registration benchmark has demonstrated that single-scale U-Net architectures, such as VoxelMorph that directly employ a spatial transformer loss, often do not generalise well beyond the cranial vault and fall short of state-of-the-art performance for abdominal or intra-patient lung registration. Here, we propose two straightforward steps that greatly reduce this gap in accuracy. First, we employ keypoint self-s…
Implicit Deformable Medical Image Registration with Learnable Kernels
Department of Computer Science Interactive Graphics and Simulation Group (IGS), University of Innsbruck, Technikerstraße 21 Innsbruck, Austria（因计算机科学系交互式图形与模拟组（IGS），因斯布鲁克大学）；Interventional Oncology-Microinvasive Therapy (SIP), Department of Radiology, Medical University Innsbruck, Innsbruck, Austria（介入肿瘤学-微侵袭疗法（SIP），放射学系，因斯布鲁克医学院）
Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；Shanghai Qi Zhi Institute（上海期智研究院）
Keypoint detection is an essential component for the object registration and alignment. In this work, we reckon keypoint detection as information compression, and force the model to distill out irrelevant points of an object. Based on this, we propose UKPGAN, a general self-supervised 3D keypoint detector where keypoints are detected so that they could reconstruct the original object shape. Two modules: GAN-based keypoint sparsity control and salient information distillation modules are proposed to locate those important keypoints. Extensive experiments show that our keypoints align well with …
Pattern Recognition Lab, FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学模式识别实验室）；Department of Ophthalmology, Universitätsklinikum Erlangen（埃尔朗根大学医院眼科）
Ophthalmological imaging utilizes different imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography. Multiple images with different modalities or acquisition times are often analyzed for the diagnosis of retinal diseases. Automatically aligning the vessel structures in the images by means of multi-modal registration can support the ophthalmologists in their work. Our method uses a convolutional neural network to extract features of the vessel structure in multi-modal retinal images. We jointly train a keypoint detection an…
Unsupervised Deep Learning-based Keypoint Localization Estimating Descriptor Matching Performance
Grupo VARPA（VARPA组）；Instituto de Investigacion Biomédica de A Coruña (INIBIC)（阿罗纳生物医学研究所（INIBIC））；Universidade da Coruña（科鲁纳大学）；Departamento de Ciencias de la Computación y Tecnologías de la Información（计算机科学与信息技术部门）
Technische Universität München（慕尼黑工业大学）；Google（谷歌）
3D Point clouds are a rich source of information that enjoy growing popularity in the vision community. However, due to the sparsity of their representation, learning models based on large point clouds is still a challenge. In this work, we introduce Graphite, a GRAPH-Induced feaTure Extraction pipeline, a simple yet powerful feature transform and keypoint detector. Graphite enables intensive down-sampling of point clouds with keypoint detection accompanied by a descriptor. We construct a generic graph-based learning scheme to describe point cloud regions and extract salient points. To this en…
Registration by Regression (RbR): a framework for interpretable and flexible atlas registration
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Danish Research Centre for Magnetic Resonance（丹麦磁共振研究中心）；Copenhagen University Hospital（哥本哈根大学医院）；Centre for Medical Image Computing（医学图像计算中心）；University College London（伦敦大学学院）；Computer Science and AI Laboratory（计算机科学与人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
Deformable Image Registration of Dark-Field Chest Radiographs for Local Lung Signal Change Assessment
Technical University of Munich（慕尼黑工业大学）；Munich Institute for Biomedical Engineering, Technical University of Munich（慕尼黑工业大学慕尼黑生物医学工程研究所）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学右岸医院）；TUM University Hospital（慕尼黑工业大学大学医院）；Philips Innovative Technologies（飞利浦创新技术公司）；Institute for Advanced Study, Technical University of Munich（慕尼黑工业大学高等研究院）
University of Utah（犹他大学）；University of Maryland, School of Medicine（马里兰大学医学院）
Radiation therapy presents a need for dynamic tracking of a target tumor volume. Fiducial markers such as implanted gold seeds have been used to gate radiation delivery but the markers are invasive and gating significantly increases treatment time. Pretreatment acquisition of a respiratory correlated 4DCT allows for determination of accurate motion tracking which is useful in treatment planning. We design a patient-specific motion subspace and a deep convolutional neural network to recover anatomical positions from a single fluoroscopic projection in real-time. We use this deep network to appr…
École Polytechnique de Montréal（蒙特利尔高等理工学院）；École de technologie supérieure（高等工程技术学院）
In this paper, an automatic method is proposed to perform image registration in visible and infrared pair of video sequences for multiple targets. In multimodal image analysis like image fusion systems, color and IR sensors are placed close to each other and capture a same scene simultaneously, but the videos are not properly aligned by default because of different fields of view, image capturing information, working principle and other camera specifications. Because the scenes are usually not planar, alignment needs to be performed continuously by extracting relevant common information. In th…
Istituto Italiano di Tecnologia（意大利理工学院）；Politecnico di Milano（米兰理工大学）；University College London（伦敦大学学院）；University College London Hospital（伦敦大学学院医院）；University Hospital Leuven（鲁汶大学医院）；Istituto "Giannina Gaslini"（詹尼娜·加斯利尼研究所）；Scuola Superiore Sant’Anna（圣安娜高等学校）
In Twin-to-Twin Transfusion Syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the two fetuses. In the current practice, TTTS is treated surgically by closing abnormal anastomoses using laser ablation. This surgery is minimally invasive and relies on fetoscopy. Limited field of view makes anastomosis identification a challenging task for the surgeon. To tackle this challenge, we propose a learning-based framework for in-vivo fetoscopy frame registration for field-of-view expansion. The novelties of this framework relies on a learn…
Deep Biomechanically-Guided Interpolation for Keypoint-Based Brain Shift Registration
LASIGE, Faculdade de Ciências da Universidade de Lisboa（里斯本大学科学学院LASIGE中心）；CRUK Cambridge Centre, University of Cambridge（剑桥大学CRUK剑桥中心）；Department of Oncology, University of Cambridge（剑桥大学肿瘤学系）；Intelligent System for Medicine Laboratory (ISML), School of Mechanical Engineering, The University of Western Australia（西澳大学机械工程学院医学智能系统实验室）；MIND Team, Inria Saclay, Université Paris-Saclay, Palaiseau, France（法国巴黎萨克利大学MIND团队）；Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（索邦大学巴黎脑研究所-ICM）
An established approach for 3D point cloud registration is to estimate the registration function from 3D keypoint correspondences. Typically, a robust technique is required to conduct the estimation, since there are false correspondences or outliers. Current 3D keypoint techniques are much less accurate than their 2D counterparts, thus they tend to produce extremely high outlier rates. A large number of putative correspondences must thus be extracted to ensure that sufficient good correspondences are available. Both factors (high outlier rates, large data sizes) however cause existing robust t…
UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Imperial College London（帝国理工学院）；Harvard Medical School（哈佛医学院）
This work investigates the use of robust optimal transport (OT) for shape matching. Specifically, we show that recent OT solvers improve both optimization-based and deep learning methods for point cloud registration, boosting accuracy at an affordable computational cost. This manuscript starts with a practical overview of modern OT theory. We then provide solutions to the main difficulties in using this framework for shape matching. Finally, we showcase the performance of transport-enhanced registration models on a wide range of challenging tasks: rigid registration for partial shapes; scene f…
