We present a generic method for assessing the quality of non-rigid registration (NRR) algorithms, that does not depend on the existence of any ground truth, but depends solely on the data itself. The data is a set of images. The output of any NRR of such a set of images is a dense correspondence across the whole set. Given such a dense correspondence, it is possible to build various generative statistical models of appearance variation across the set. We show that evaluating the quality of the registration can be mapped to the problem of evaluating the quality of the resultant statistical model. The quality of the model entails a comparison between the model and the image data that was used to construct it. It should be noted that this approach does not depend on the specifics of the registration algorithm used (i.e., whether a groupwise or pairwise algorithm was used to register the set of images), or on the specifics of the modelling approach used.
We derive an index of image model specificity that can be used to assess image model quality, and hence the quality of registration. This approach is validated by comparing our assessment of registration quality with that derived from ground truth anatomical labeling. We demonstrate that our approach is capable of assessing NRR reliably without ground truth. Finally, to demonstrate the practicality of our method, different NRR algorithms -- both pairwise and groupwise -- are compared in terms of their performance on 3D MR brain data.
Hannover Medical School（汉诺威医学院）；Leibniz-Fachhochschule School of Business（莱布尼茨应用科学大学商学院）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Biomedical Research in Endstage and Obstructive Lung Disease Hannover (BREATH), Member of the German Center for Lung Research (DZL)（汉诺威终末期与阻塞性肺病生物医学研究（BREATH），德国肺研究中心（DZL）成员）
Registration of histological serial sections is a challenging task. Serial sections exhibit distortions and damage from sectioning. Missing information on how the tissue looked before cutting makes a realistic validation of 2D registrations extremely difficult. This work proposes methods for ground-truth-based evaluation of registrations. Firstly, we present a methodology to generate test data for registrations. We distort an innately registered image stack in the manner similar to the cutting distortion of serial sections. Test cases are generated from existing 3D data sets, thus the ground t…
Yale University（耶鲁大学）；Yale School of Medicine（耶鲁大学医学院）；University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Chinese Academy of Sciences（中国科学院）
Multimodal image registration has many applications in diagnostic medical imaging and image-guided interventions, such as Transcatheter Arterial Chemoembolization (TACE) of liver cancer guided by intraprocedural CBCT and pre-operative MR. The ability to register peri-procedurally acquired diagnostic images into the intraprocedural environment can potentially improve the intra-procedural tumor targeting, which will significantly improve therapeutic outcomes. However, the intra-procedural CBCT often suffers from suboptimal image quality due to lack of signal calibration for Hounsfield unit, limi…
Seoul National University（首尔国立大学）；Soongsil University（崇实大学）
Accurate three-dimensional perception is a fundamental task in several computer vision applications. Recently, commercial RGB-depth (RGB-D) cameras have been widely adopted as single-view depth-sensing devices owing to their efficient depth-sensing abilities. However, the depth quality of most RGB-D sensors remains insufficient owing to the inherent noise from a single-view environment. Recently, several studies have focused on the single-view depth enhancement of RGB-D cameras. Recent research has proposed deep-learning-based approaches that typically train networks using high-quality supervi…
AI-assisted surgeries have drawn the attention of the medical image research community due to their real-world impact on improving surgery success rates. For image-guided surgeries, such as Cochlear Implants (CIs), accurate object segmentation can provide useful information for surgeons before an operation. Recently published image segmentation methods that leverage machine learning usually rely on a large number of manually predefined ground truth labels. However, it is a laborious and time-consuming task to prepare the dataset. This paper presents a novel technique using a self-supervised 3D…
National Aerospace University（国立航空航天大学）；University of Rennes 1（雷恩第一大学）
This paper focuses on potential accuracy of remote sensing images registration. We investigate how this accuracy can be estimated without ground truth available and used to improve registration quality of mono- and multi-modal pair of images. At the local scale of image fragments, the Cramer-Rao lower bound (CRLB) on registration error is estimated for each local correspondence between coarsely registered pair of images. This CRLB is defined by local image texture and noise properties. Opposite to the standard approach, where registration accuracy is only evaluated at the output of the registr…
Deformable image registration is a fundamental task in medical image analysis, aiming to establish a dense and non-linear correspondence between a pair of images. Previous deep-learning studies usually employ supervised neural networks to directly learn the spatial transformation from one image to another, requiring task-specific ground-truth registration for model training. Due to the difficulty in collecting precise ground-truth registration, implementation of these supervised methods is practically challenging. Although several unsupervised networks have been recently developed, these metho…
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a reg…
Aachen Institute for Advanced Study in Computational Engineering Science (AICES), RWTH Aachen University（亚琛工业大学计算工程科学高级研究院（AICES））；Peter L. Reichertz Institute for Medical Informatics, University of Braunschweig – Institute of Technology and Hannover Medical School（布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所）
Region of interest (ROI) alignment in medical images plays a crucial role in diagnostics, procedure planning, treatment, and follow-up. Frequently, a model is represented as triangulated mesh while the patient data is provided from CAT scanners as pixel or voxel data. Previously, we presented a 2D method for curve-to-pixel registration. This paper contributes (i) a general mesh-to-raster (M2R) framework to register ROIs in multi-modal images; (ii) a 3D surface-to-voxel application, and (iii) a comprehensive quantitative evaluation in 2D using ground truth provided by the simultaneous truth and…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Northwestern Polytechnical University（西北工业大学）；Korea University（高丽大学）
In this paper, we propose a deep learning approach for image registration by predicting deformation from image appearance. Since obtaining ground-truth deformation fields for training can be challenging, we design a fully convolutional network that is subject to dual-guidance: (1) Coarse guidance using deformation fields obtained by an existing registration method; and (2) Fine guidance using image similarity. The latter guidance helps avoid overly relying on the supervision from the training deformation fields, which could be inaccurate. For effective training, we further improve the deep con…
Deformable registration is one of the most challenging task in the field of medical image analysis, especially for the alignment between different sequences and modalities. In this paper, a non-rigid registration method is proposed for 3D medical images leveraging unsupervised learning. To the best of our knowledge, this is the first attempt to introduce gradient loss into deep-learning-based registration. The proposed gradient loss is robust across sequences and modals for large deformation. Besides, adversarial learning approach is used to transfer multi-modal similarity to mono-modal simila…
We present a fast learning-based algorithm for deformable, pairwise 3D medical image registration. Current registration methods optimize an objective function independently for each pair of images, which can be time-consuming for large data. We define registration as a parametric function, and optimize its parameters given a set of images from a collection of interest. Given a new pair of scans, we can quickly compute a registration field by directly evaluating the function using the learned parameters. We model this function using a convolutional neural network (CNN), and use a spatial transf…
Rheinische Friedrich-Wilhelms-Universität Bonn（波恩大学）；Hôpitaux Universitaires de Genève（日内瓦大学医院）
Brain shift, i.e. the change in configuration of the brain after opening the dura mater, is a key problem in neuronavigation. We present an approach to co-register intra-operative microscope images with pre-operative MRI to adapt and optimize intra-operative neuronavigation. The tools are a robust classification of sulci on MRI extracted cortical surfaces, guided user marking of most prominent sulci on a microscope image, and the actual variational registration method with a fidelity energy for 3D deformations of the cortical surface combined with a higher order, linear elastica type prior ene…
Dense registration of fingerprints is a challenging task due to elastic skin distortion, low image quality, and self-similarity of ridge pattern. To overcome the limitation of handcraft features, we propose to train an end-to-end network to directly output pixel-wise displacement field between two fingerprints. The proposed network includes a siamese network for feature embedding, and a following encoder-decoder network for regressing displacement field. By applying displacement fields reliably estimated by tracing high quality fingerprint videos to challenging fingerprints, we synthesize a la…
The Hong Kong University of Science and Technology（香港科技大学）
Diffeomorphic deformable image registration is crucial in many medical image studies, as it offers unique, special properties including topology preservation and invertibility of the transformation. Recent deep learning-based deformable image registration methods achieve fast image registration by leveraging a convolutional neural network (CNN) to learn the spatial transformation from the synthetic ground truth or the similarity metric. However, these approaches often ignore the topology preservation of the transformation and the smoothness of the transformation which is enforced by a global s…
NYU Multimedia and Visual Computing Lab, NYU Tandon（纽约大学坦登工程学院多媒体与视觉计算实验室）；NYU Abu Dhabi（纽约大学阿布扎比分校）；Department of Electrical and Computer Engineering, NYU Tandon（纽约大学坦登工程学院电气与计算机工程系）；Department of Electrical and Computer Engineering, NYU Abu Dhabi（纽约大学阿布扎比分校电气与计算机工程系）
We propose a self-supervised method for partial point set registration. While recent proposed learning-based methods have achieved impressive registration performance on the full shape observations, these methods mostly suffer from performance degradation when dealing with partial shapes. To bridge the performance gaps between partial point set registration with full point set registration, we proposed to incorporate a shape completion network to benefit the registration process. To achieve this, we design a latent code for each pair of shapes, which can be regarded as a geometric encoding of …
Instituto Tecnológico y de Estudios Superiores de Monterrey（蒙特雷技术与高等教育学院）
Early tumor detection is key in reducing the number of breast cancer death and screening mammography is one of the most widely available and reliable method for early detection. However, it is difficult for the radiologist to process with the same attention each case, due the large amount of images to be read. Computer aided detection (CADe) systems improve tumor detection rate; but the current efficiency of these systems is not yet adequate and the correct interpretation of CADe outputs requires expert human intervention. Computer aided diagnosis systems (CADx) are being designed to improve c…
In ophthalmological imaging, multiple imaging systems, such as color fundus, infrared, fluorescein angiography, optical coherence tomography (OCT) or OCT angiography, are often involved to make a diagnosis of retinal disease. Multi-modal retinal registration techniques can assist ophthalmologists by providing a pixel-based comparison of aligned vessel structures in images from different modalities or acquisition times. To this end, we propose an end-to-end trainable deep learning method for multi-modal retinal image registration. Our method extracts convolutional features from the vessel struc…
In this paper a fast triangular mesh based registration method is proposed. Having Template and Reference images as inputs, the template image is triangulated using a content adaptive mesh generation algorithm. Considering the pixel values at mesh nodes, interpolated using spline interpolation method for both of the images, the energy functional needed for image registration is minimized. The minimization process was achieved using a mesh based discretization of the distance measure and regularization term which resulted in a sparse system of linear equations, which due to the smaller size in …
South China University of Technology（华南理工大学）；Rochester Institute of Technology（罗切斯特理工学院）
In clinical practice, well-aligned multi-modal images, such as Magnetic Resonance (MR) and Computed Tomography (CT), together can provide complementary information for image-guided therapies. Multi-modal image registration is essential for the accurate alignment of these multi-modal images. However, it remains a very challenging task due to complicated and unknown spatial correspondence between different modalities. In this paper, we propose a novel translation-based unsupervised deformable image registration approach to convert the multi-modal registration problem to a mono-modal one. Specifi…
On-the-Fly Guidance Training for Medical Image Registration
University of Leeds（利兹大学）；University of California, Irvine（加州大学欧文分校）；Tongji University（同济大学）；Huazhong University of Science and Technology（华中科技大学）；Fudan Univeristy（复旦大学）；University of California, San Diego（加州大学圣迭戈分校）
Imperial College London（帝国理工学院）；Siemens Healthineers（西门子医疗）
We propose a fully unsupervised multi-modal deformable image registration method (UMDIR), which does not require any ground truth deformation fields or any aligned multi-modal image pairs during training. Multi-modal registration is a key problem in many medical image analysis applications. It is very challenging due to complicated and unknown relationships between different modalities. In this paper, we propose an unsupervised learning approach to reduce the multi-modal registration problem to a mono-modal one through image disentangling. In particular, we decompose images of both modalities …
University College London（伦敦大学学院）；Verdure Imaging, Inc.（Verdure成像公司）；Queen’s University（女王大学）
Free Point Transformer (FPT) has been proposed as a data-driven, non-rigid point set registration approach using deep neural networks. As FPT does not assume constraints based on point vicinity or correspondence, it may be trained simply and in a flexible manner by minimizing an unsupervised loss based on the Chamfer Distance. This makes FPT amenable to real-world medical imaging applications where ground-truth deformations may be infeasible to obtain, or in scenarios where only a varying degree of completeness in the point sets to be aligned is available. To test the limit of the corresponden…
Deformable image registration is able to achieve fast and accurate alignment between a pair of images and thus plays an important role in many medical image studies. The current deep learning (DL)-based image registration approaches directly learn the spatial transformation from one image to another by leveraging a convolutional neural network, requiring ground truth or similarity metric. Nevertheless, these methods only use a global similarity energy function to evaluate the similarity of a pair of images, which ignores the similarity of regions of interest (ROIs) within images. Moreover, DL-…
It is not easy when evaluating 3D mapping performance because existing metrics require ground truth data that can only be collected with special instruments. In this paper, we propose a metric, dense map posterior (DMP), for this evaluation. It can work without any ground truth data. Instead, it calculates a comparable value, reflecting a map posterior probability, from dense point cloud observations. In our experiments, the proposed DMP is benchmarked against ground truth-based metrics. Results show that DMP can provide a similar evaluation capability. The proposed metric makes evaluating dif…
Università degli Studi di Milano - Bicocca（米兰比可卡大学）；Albert Ludwigs Universität Freiburg（阿尔伯特·路德维希·弗莱堡大学）；Universidad de Alcala de Henares（阿尔卡拉大学）
Point clouds registration is a fundamental step of many point clouds processing pipelines; however, most algorithms are tested on data that are collected ad-hoc and not shared with the research community. These data often cover only a very limited set of use cases; therefore, the results cannot be generalised. Public datasets proposed until now, taken individually, cover only a few kinds of environment and mostly a single sensor. For these reasons, we developed a benchmark, for localization and mapping applications, using multiple publicly available datasets. In this way, we are able to cover …
Medical image registration and segmentation are critical tasks for several clinical procedures. Manual realisation of those tasks is time-consuming and the quality is highly dependent on the level of expertise of the physician. To mitigate that laborious task, automatic tools have been developed where the majority of solutions are supervised techniques. However, in medical domain, the strong assumption of having a well-representative ground truth is far from being realistic. To overcome this challenge, unsupervised techniques have been investigated. However, they are still limited in performan…
Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion
College of Intelligence and Computing（智能与计算学院）；Tianjin University（天津大学）；School of Computer Science and Technology（计算机科学与技术学校）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
University of South Australia（南澳大学）；Defence Science and Technology Group（国防科技集团）
One of the challenges in evaluating multi-object video detection, tracking and classification systems is having publically available data sets with which to compare different systems. However, the measures of performance for tracking and classification are different. Data sets that are suitable for evaluating tracking systems may not be appropriate for classification. Tracking video data sets typically only have ground truth track IDs, while classification video data sets only have ground truth class-label IDs. The former identifies the same object over multiple frames, while the latter identi…
Purpose: Segmentation of organs-at-risk (OARs) is a bottleneck in current radiation oncology pipelines and is often time consuming and labor intensive. In this paper, we propose an atlas-based semi-supervised registration algorithm to generate accurate segmentations of OARs for which there are ground truth contours and rough segmentations of all other OARs in the atlas. To the best of our knowledge, this is the first study to use learning-based registration methods for the segmentation of head and neck patients and demonstrate its utility in clinical applications. Methods: Our algorithm cascad…
A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints
Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）；Weill Cornell Medical School（威尔康奈尔医学院）；Iterative Scopes（Iterative Scopes公司）；Massachusetts Institute of Technology（麻省理工学院）；Massachusetts General Hospital（麻省总医院）
Radboud University Medical Center（拉德堡德大学医学中心）；University of Leeds（利兹大学）；Sichuan University（四川大学）；University of North Carolina（北卡罗来纳大学）
Unsupervised learning strategy is widely adopted by the deformable registration models due to the lack of ground truth of deformation fields. These models typically depend on the intensity-based similarity loss to obtain the learning convergence. Despite the success, such dependence is insufficient. For the deformable registration of mono-modality image, well-aligned two images not only have indistinguishable intensity differences, but also are close in the statistical distribution and the boundary areas. Considering that well-designed loss functions can facilitate a learning model into a desi…
CAD-to-CT Registration of Cylindrical Objects via Ellipse-Based Axis Estimation
Department of Complex Systems, National Centre for Nuclear Research（复杂系统系，国家核研究中心）；ImagineRT sp. z o.o.（ImagineRT公司）；National Centre for Nuclear Research（国家核研究中心）
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）
Deformable image registration between Computed Tomography (CT) images and Magnetic Resonance (MR) imaging is essential for many image-guided therapies. In this paper, we propose a novel translation-based unsupervised deformable image registration method. Distinct from other translation-based methods that attempt to convert the multimodal problem (e.g., CT-to-MR) into a unimodal problem (e.g., MR-to-MR) via image-to-image translation, our method leverages the deformation fields estimated from both: (i) the translated MR image and (ii) the original CT image in a dual-stream fashion, and automati…
Evaluation of Deformable Image Registration under Alignment-Regularity Trade-off
Technical University of Munich (TUM)（慕尼黑技术大学）；TUM University Hospital, Munich, Germany（慕尼黑技术大学医院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）
University College London（伦敦大学学院）；UCL Institute of Neurology（伦敦大学学院神经病学研究所）；Wellcome / EPSRC Centre for Interventional and Surgical Sciences, UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）
Image registration between histology and magnetic resonance imaging (MRI) is a challenging task due to differences in structural content and contrast. Too thick and wide specimens cannot be processed all at once and must be cut into smaller pieces. This dramatically increases the complexity of the problem, since each piece should be individually and manually pre-aligned. To the best of our knowledge, no automatic method can reliably locate such piece of tissue within its respective whole in the MRI slice, and align it without any prior information. We propose here a novel automatic approach to…
Deformable Image Registration (DIR) of MR and CT images is one of the most challenging registration task, due to the inherent structural differences of the modalities and the missing dense ground truth. Recently cycle Generative Adversarial Networks (cycle-GANs) have been used to learn the intensity relationship between these 2 modalities for unpaired brain data. Yet its usefulness for DIR was not assessed. In this study we evaluate the DIR performance for thoracic and abdominal organs after synthesis by cycle-GAN. We show that geometric changes, which differentiate the two populations (e.g. i…
New York University（纽约大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）；NYU Tandon School of Engineering（纽约大学坦登工程学院）
Point cloud registration is the process of aligning a pair of point sets via searching for a geometric transformation. Recent works leverage the power of deep learning for registering a pair of point sets. However, unfortunately, deep learning models often require a large number of ground truth labels for training. Moreover, for a pair of source and target point sets, existing deep learning mechanisms require explicitly designed encoders to extract both deep spatial features from unstructured point clouds and their spatial correlation representation, which is further fed to a decoder to regres…
Computer Science and Artificial Intelligence Lab, MIT（麻省理工学院计算机科学与人工智能实验室）；Meinig School of Biomedical Engineering, Cornell University（康奈尔大学迈尼格生物医学工程学院）；Martinos Center for Biomedical Imaging, MGH, HMS（麻省总医院哈佛医学院马蒂诺生物医学成像中心）
We present VoxelMorph, a fast learning-based framework for deformable, pairwise medical image registration. Traditional registration methods optimize an objective function for each pair of images, which can be time-consuming for large datasets or rich deformation models. In contrast to this approach, and building on recent learning-based methods, we formulate registration as a function that maps an input image pair to a deformation field that aligns these images. We parameterize the function via a convolutional neural network (CNN), and optimize the parameters of the neural network on a set of…
NASA Goddard Space Flight Center（美国国家航空航天局戈达德太空飞行中心）；Ball Aerospace and Technologies Corp.（鲍尔航空航天技术公司）；Adnet Systems Inc.（Adnet系统公司）
Image registration, or alignment of two or more images covering the same scenes or objects, is of great interest in many disciplines such as remote sensing, medical imaging, astronomy, and computer vision. In this paper, we introduce a new application of image registration algorithms. We demonstrate how through a wavelet based image registration algorithm, engineers can evaluate stability of Micro-Electro-Mechanical Systems (MEMS). In particular, we applied image registration algorithms to assess alignment stability of the MicroShutters Subsystem (MSS) of the Near Infrared Spectrograph (NIRSpe…
Be Careful When Evaluating Explanations Regarding Ground Truth
University of Warsaw（华沙大学）；Warsaw University of Technology（华沙理工大学）；University of Natural Resources and Life Sciences Vienna（维也纳自然资源与生命科学大学）；Medical University of Graz（格拉茨医科大学）
Unsupervised Segmentation of Fetal Brain MRI using Deep Learning Cascaded Registration
Universitat Pompeu Fabra（庞培法布拉大学）；BCNatal；Institución Catalana de Investigación y Estudios Avanzados (ICREA)（加泰罗尼亚高级研究与发展研究院）；Centre for Biomedical Research on Rare Diseases (CIBERER)（罕见疾病生物医学研究中心）；European Commission, Joint Research Centre (JRC)（欧盟委员会联合研究中心）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Auris Health, Inc.（奥里斯健康公司）；Johns Hopkins Medicine（约翰斯·霍普金斯医学院）；Nara Institute of Science and Technology（奈良科学技术研究所）；University of Texas（德克萨斯大学）；Texas Orthopedics（德克萨斯骨科医院）
Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. We propose a method for fully automatic registration using annotations produced by a neural network. Neural networks are trained to simultaneously segment anatomy and ide…
