Understanding the uncertainty inherent in deep learning-based image registration models has been an ongoing area of research. Existing methods have been developed to quantify both transformation and appearance uncertainties related to the registration process, elucidating areas where the model may exhibit ambiguity regarding the generated deformation. However, our study reveals that neither uncertainty effectively estimates the potential errors when the registration model is used for label propagation. Here, we propose a novel framework to concurrently estimate both the epistemic and aleatoric segmentation uncertainties for image registration. To this end, we implement a compact deep neural network (DNN) designed to transform the appearance discrepancy in the warping into aleatoric segmentation uncertainty by minimizing a negative log-likelihood loss function. Furthermore, we present epistemic segmentation uncertainty within the label propagation process as the entropy of the propagated labels. By introducing segmentation uncertainty along with existing methods for estimating registration uncertainty, we offer vital insights into the potential uncertainties at different stages of image registration. We validated our proposed framework using publicly available datasets, and the results prove that the segmentation uncertainties estimated with the proposed method correlate well with errors in label propagation, all while achieving superior registration performance.
ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究中心）
Image registration of liver dynamic contrast-enhanced computed tomography (DCE-CT) is crucial for diagnosis and image-guided surgical planning of liver cancer. However, intensity variations due to the flow of contrast agents combined with complex spatial motion induced by respiration brings great challenge to existing intensity-based registration methods. To address these problems, we propose a novel structure-aware registration method by incorporating structural information of related organs with segmentation-guided deep registration network. Existing segmentation-guided registration methods …
Deep learning models for semantic segmentation are able to learn powerful representations for pixel-wise predictions, but are sensitive to noise at test time and do not guarantee a plausible topology. Image registration models on the other hand are able to warp known topologies to target images as a means of segmentation, but typically require large amounts of training data, and have not widely been benchmarked against pixel-wise segmentation models. We propose Atlas-ISTN, a framework that jointly learns segmentation and registration on 2D and 3D image data, and constructs a population-derived…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Erasmus MC - University Medical Center Rotterdam（伊拉斯谟医学中心——鹿特丹大学医学中心）；University of Copenhagen（哥本哈根大学）；Massachusetts Institute of Technology（麻省理工学院）
Registration is a core component of many imaging pipelines. In case of clinical scans, with lower resolution and sometimes substantial motion artifacts, registration can produce poor results. Visual assessment of registration quality in large clinical datasets is inefficient. In this work, we propose to automatically assess the quality of registration to an atlas in clinical FLAIR MRI scans of the brain. The method consists of automatically segmenting the ventricles of a given scan using a neural network, and comparing the segmentation to the atlas' ventricles propagated to image space. We use…
University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
Deep convolutional neural networks (CNNs) are state-of-the-art for semantic image segmentation, but typically require many labeled training samples. Obtaining 3D segmentations of medical images for supervised training is difficult and labor intensive. Motivated by classical approaches for joint segmentation and registration we therefore propose a deep learning framework that jointly learns networks for image registration and image segmentation. In contrast to previous work on deep unsupervised image registration, which showed the benefit of weak supervision via image segmentations, our approac…
TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Research
University of Strasbourg（斯特拉斯堡大学）；Research Institute against Digestive Cancer (IRCAD)（消化癌症研究所（IRCAD））；Conservatoire National des Arts et Métiers (CNAM)（法国国立工艺学院（CNAM））；Sorbonne University（索邦大学）；Clinique Sainte-Anne（圣安妮诊所）；Clinique Saint Augustin（圣奥古斯丁诊所）；Institute of Image-Guided Surgery (IHU)（图像引导手术研究所（IHU））
Monash eResearch Center, Monash University（莫纳什大学莫纳什电子研究中心）；AirDoc Research Australia（澳大利亚AirDoc研究公司）；IBM Research Australia（IBM澳大利亚研究院）
Registration is an important task in automated medical image analysis. Although deep learning (DL) based image registration methods out perform time consuming conventional approaches, they are heavily dependent on training data and do not generalize well for new images types. We present a DL based approach that can register an image pair which is different from the training images. This is achieved by training generative adversarial networks (GANs) in combination with segmentation information and transfer learning. Experiments on chest Xray and brain MR images show that our method gives better…
University College London（伦敦大学学院）；University of Oxford（牛津大学）；Siemens Healthineers（西门子医疗）；King’s College London（伦敦国王学院）
Classical pairwise image registration methods search for a spatial transformation that optimises a numerical measure that indicates how well a pair of moving and fixed images are aligned. Current learning-based registration methods have adopted the same paradigm and typically predict, for any new input image pair, dense correspondences in the form of a dense displacement field or parameters of a spatial transformation model. However, in many applications of registration, the spatial transformation itself is only required to propagate points or regions of interest (ROIs). In such cases, detaile…
Multi-atlas segmentation (MAS) is a promising framework for medical image segmentation. Generally, MAS methods register multiple atlases, i.e., medical images with corresponding labels, to a target image; and the transformed atlas labels can be combined to generate target segmentation via label fusion schemes. Many conventional MAS methods employed the atlases from the same modality as the target image. However, the number of atlases with the same modality may be limited or even missing in many clinical applications. Besides, conventional MAS methods suffer from the computational burden of reg…
The implementation of automatic image registration is still difficult in various applications. In this paper, an automatic image registration approach through line-support region segmentation and geometrical outlier removal (ALRS-GOR) is proposed. This new approach is designed to address the problems associated with the registration of images with affine deformations and inconsistent content, such as remote sensing images with different spectral content or noise interference, or map images with inconsistent annotations. To begin with, line-support regions, namely a straight region whose points…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；Department of Automation, Tsinghua University（清华大学自动化系）；Center for Advanced Intelligence Project, RIKEN（理化学研究所先进智能项目中心）
In image-guided neurosurgery, current commercial systems usually provide only rigid registration, partly because it is harder to predict, validate and understand non-rigid registration error. For instance, when surgeons see a discrepancy in aligned image features, they may not be able to distinguish between registration error and actual tissue deformation caused by tumor resection. In this case, the spatial distribution of registration error could help them make more informed decisions, e.g., ignoring the registration where the estimated error is high. However, error estimates are difficult to…
Cluster of Excellence – ML for Science（卓越集群——科学机器学习）；University of Tübingen（蒂宾根大学）；Faculty of Health Sciences and Medicine, University of Lucerne（卢塞恩大学健康科学与医学院）；University of Lübeck（吕贝克大学）
The Chinese University of Hong Kong（香港中文大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Tencent Jarvis Lab（腾讯医疗健康（深圳）有限公司（腾讯医疗AI实验室/腾讯天衍实验室））；Tsinghua University（清华大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
In order to tackle the difficulty associated with the ill-posed nature of the image registration problem, regularization is often used to constrain the solution space. For most learning-based registration approaches, the regularization usually has a fixed weight and only constrains the spatial transformation. Such convention has two limitations: (i) Besides the laborious grid search for the optimal fixed weight, the regularization strength of a specific image pair should be associated with the content of the images, thus the "one value fits all" training scheme is not ideal; (ii) Only spatiall…
Image registration is one of the most challenging problems in medical image analysis. In the recent years, deep learning based approaches became quite popular, providing fast and performing registration strategies. In this short paper, we summarise our work presented on Learn2Reg challenge 2020. The main contributions of our work rely on (i) a symmetric formulation, predicting the transformations from source to target and from target to source simultaneously, enforcing the trained representations to be similar and (ii) integration of variety of publicly available datasets used both for pretrai…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）；School of Computing, Queen‘s University（女王大学计算学院）；School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；Computing Science Department, University of Alberta（阿尔伯塔大学计算科学系）；Ecole de Technologie Superieure（高等技术学院）；Center for Advanced Intelligence Project, RIKEN（理化学研究所高级智能项目中心）
Estimating the uncertainty in (probabilistic) image registration enables, e.g., surgeons to assess the operative risk based on the trustworthiness of the registered image data. If surgeons receive inaccurately calculated registration uncertainty and misplace unwarranted confidence in the alignment solutions, severe consequences may result. For probabilistic image registration (PIR), the predominant way to quantify the registration uncertainty is using summary statistics of the distribution of transformation parameters. The majority of existing research focuses on trying out different summary s…
Imperial College London（帝国理工学院）；Technische Universität München（慕尼黑工业大学）；Sharif University of Technology（谢里夫理工大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）
We develop a new Bayesian model for non-rigid registration of three-dimensional medical images, with a focus on uncertainty quantification. Probabilistic registration of large images with calibrated uncertainty estimates is difficult for both computational and modelling reasons. To address the computational issues, we explore connections between the Markov chain Monte Carlo by backpropagation and the variational inference by backpropagation frameworks, in order to efficiently draw samples from the posterior distribution of transformation parameters. To address the modelling issues, we formulat…
Current deep-learning-based registration algorithms often exploit intensity-based similarity measures as the loss function, where dense correspondence between a pair of moving and fixed images is optimized through backpropagation during training. However, intensity-based metrics can be misleading when the assumption of intensity class correspondence is violated, especially in cross-modality or contrast-enhanced images. Moreover, existing learning-based registration methods are predominantly applicable to pairwise registration and are rarely extended to groupwise registration or simultaneous re…
Fuzhou University（福州大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；King’s College London（伦敦国王学院）
Both image registration and label fusion in the multi-atlas segmentation (MAS) rely on the intensity similarity between target and atlas images. However, such similarity can be problematic when target and atlas images are acquired using different imaging protocols. High-level structure information can provide reliable similarity measurement for cross-modality images when cooperating with deep neural networks (DNNs). This work presents a new MAS framework for cross-modality images, where both image registration and label fusion are achieved by DNNs. For image registration, we propose a consiste…
This paper presents a novel probabilistic voxel selection strategy for medical image registration in time-sensitive contexts, where the goal is aggressive voxel sampling (e.g. using less than 1% of the total number) while maintaining registration accuracy and low failure rate. We develop a Bayesian framework whereby, first, a voxel sampling probability field (VSPF) is built based on the uncertainty on the transformation parameters. We then describe a practical, multi-scale registration algorithm, where, at each optimization iteration, different voxel subsets are sampled based on the VSPF. The …
Harvard Medical School（哈佛医学院）；INRIA Sophia Antipolis-Méditerranée（法国国家信息与自动化研究所索菲亚-安提波利斯-地中海中心）；University of North Carolina（北卡罗来纳大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a novel mathematical framework for representing uncertainty in large deformation diffeomorphic image registration. The Bayesian posterior distribution over the deformations aligning a moving and a fixed image is approximated via a variational formulation. A stochastic differential equation (SDE) modeling the deformations as the evolution of a time-varying velocity field leads to a prior density over deformations in the form of a Gaussian process. This permits estimating the full posterior distribution in order to represent uncertainty, in contrast to methods in which the po…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Whirlpool（惠而浦）
We present WSSAMNet, a weakly supervised method for medical image registration. Ours is a two step method, with the first step being the computation of segmentation masks of the fixed and moving volumes. These masks are then used to attend to the input volume, which are then provided as inputs to a registration network in the second step. The registration network computes the deformation field to perform the alignment between the fixed and the moving volumes. We study the effectiveness of our technique on the BraTSReg challenge data against ANTs and VoxelMorph, where we demonstrate that our me…
University College London（伦敦大学学院）；Massachusetts General Hospital（马萨诸塞总医院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Landmark correspondences are a widely used type of gold standard in image registration. However, the manual placement of corresponding points is subject to high inter-user variability in the chosen annotated locations and in the interpretation of visual ambiguities. In this paper, we introduce a principled strategy for the construction of a gold standard in deformable registration. Our framework: (i) iteratively suggests the most informative location to annotate next, taking into account its redundancy with previous annotations; (ii) extends traditional pointwise annotations by accounting for …
Variational U-Net with Local Alignment for Joint Tumor Extraction and Registration (VALOR-Net) of Breast MRI Data Acquired at Two Different Field Strengths
University of Peshawar（白沙瓦大学）；Danube Private University（多瑙河私立大学）；University of Engineering & Technology, Peshawar（白沙瓦工程与技术大学）；Medical University of Vienna（维也纳医科大学）；Karl Landsteiner Society（卡尔·兰施泰纳学会）；University of Cambridge（剑桥大学）
ShanghaiTech University（上海科技大学）；University of Zürich（苏黎世大学）；ETH Zürich（苏黎世联邦理工学院）
Multi-spectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) that offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of multi-modal images remains challenging, chiefly due to the entirely different image contrast rendered by these modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation …
Fraunhofer MEVIS（弗劳恩霍夫医学图像计算研究所）；Radboud UMC（拉德堡德大学医学中心）；University of Lübeck（吕贝克大学）
While deep learning has achieved significant advances in accuracy for medical image segmentation, its benefits for deformable image registration have so far remained limited to reduced computation times. Previous work has either focused on replacing the iterative optimization of distance and smoothness terms with CNN-layers or using supervised approaches driven by labels. Our method is the first to combine the complementary strengths of global semantic information (represented by segmentation labels) and local distance metrics that help align surrounding structures. We demonstrate significant …
Leiden University Medical Center（莱顿大学医学中心）；Leiden Institute of Advanced Computer Science（莱顿高级计算机科学研究所）
Medical image registration and segmentation are two of the most frequent tasks in medical image analysis. As these tasks are complementary and correlated, it would be beneficial to apply them simultaneously in a joint manner. In this paper, we formulate registration and segmentation as a joint problem via a Multi-Task Learning (MTL) setting, allowing these tasks to leverage their strengths and mitigate their weaknesses through the sharing of beneficial information. We propose to merge these tasks not only on the loss level, but on the architectural level as well. We studied this approach in th…
University of Magdeburg（马格德堡大学）；University Hospital Magdeburg（马格德堡大学医院）
Recent studies demonstrated the eligibility of convolutional neural networks (CNNs) for solving the image registration problem. CNNs enable faster transformation estimation and greater generalization capability needed for better support during medical interventions. Conventional fully-supervised training requires a lot of high-quality ground truth data such as voxel-to-voxel transformations, which typically are attained in a too tedious and error-prone manner. In our work, we use weakly-supervised learning, which optimizes the model indirectly only via segmentation masks that are a more access…
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）
Cone-beam CT (CBCT)-based online adaptive radiotherapy calls for accurate auto-segmentation to reduce the time cost for physicians to edit contours. However, deep learning (DL)-based direct segmentation of CBCT images is a challenging task, mainly due to the poor image quality and lack of well-labelled large training datasets. Deformable image registration (DIR) is often used to propagate the manual contours on the planning CT (pCT) of the same patient to CBCT. In this work, we undertake solving the problems mentioned above with the assistance of DIR. Our method consists of three main componen…
Robust deformable image registration using cycle-consistent implicit representations
Amsterdam UMC location University of Amsterdam（阿姆斯特丹大学医学中心（阿姆斯特丹大学））；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Amsterdam Gastroenterology Endocrinology Metabolism（阿姆斯特丹胃肠病学与内分泌代谢研究所）；Cancer Center Amsterdam（阿姆斯特丹癌症中心）
Athinoula A. Martinos Center for Biomedical Imaging（阿西努拉·A·马蒂诺斯生物医学成像中心）；Massachusetts Institute of Technology（麻省理工学院）
In recent years, learning-based image registration methods have gradually moved away from direct supervision with target warps to instead use self-supervision, with excellent results in several registration benchmarks. These approaches utilize a loss function that penalizes the intensity differences between the fixed and moving images, along with a suitable regularizer on the deformation. In this paper, we argue that the relative failure of supervised registration approaches can in part be blamed on the use of regular U-Nets, which are jointly tasked with feature extraction, feature matching, …
Primitive Simultaneous Optimization of Similarity Metrics for Image Registration
Technical University of Munich（慕尼黑工业大学）；University of Zurich（苏黎世大学）；Klinikum rechts der Isar（伊萨尔河右岸医院）；TranslaTUM - Central Institute for Translational Cancer Research（TranslaTUM——转化癌症研究中央研究所）
CortexMorph: fast cortical thickness estimation via diffeomorphic registration using VoxelMorph
Support Center for Advanced Neuroimaging (SCAN)（高级神经影像支持中心（SCAN））；University Institute of Diagnostic and Interventional Neuroradiology（诊断与介入神经放射学大学研究所）；Inselspital, Bern University Hospital（伯尔尼大学医院 Inselspital）
We introduce a fluid-based image augmentation method for medical image analysis. In contrast to existing methods, our framework generates anatomically meaningful images via interpolation from the geodesic subspace underlying given samples. Our approach consists of three steps: 1) given a source image and a set of target images, we construct a geodesic subspace using the Large Deformation Diffeomorphic Metric Mapping (LDDMM) model; 2) we sample transformations from the resulting geodesic subspace; 3) we obtain deformed images and segmentations via interpolation. Experiments on brain (LPBA) and …
University College London（伦敦大学学院）；InstaDeep；University College London Hospital NHS Foundation Trust（伦敦大学学院医院NHS基金会信托）；University of Oxford（牛津大学）
Morphological analysis of longitudinal MR images plays a key role in monitoring disease progression for prostate cancer patients, who are placed under an active surveillance program. In this paper, we describe a learning-based image registration algorithm to quantify changes on regions of interest between a pair of images from the same patient, acquired at two different time points. Combining intensity-based similarity and gland segmentation as weak supervision, the population-data-trained registration networks significantly lowered the target registration errors (TREs) on holdout patient data…
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
Robarts Research Institute（罗巴茨研究所）；Western University（韦仕敦大学）
Optimization techniques have been widely used in deformable registration, allowing for the incorporation of similarity metrics with regularization mechanisms. These regularization mechanisms are designed to mitigate the effects of trivial solutions to ill-posed registration problems and to otherwise ensure the resulting deformation fields are well-behaved. This paper introduces a novel deformable registration algorithm, RANCOR, which uses iterative convexification to address deformable registration problems under total-variation regularization. Initial comparative results against four state-of…
A semantic feature extraction method for multitemporal high resolution aerial image registration is proposed in this paper. These features encode properties or information about temporally invariant objects such as roads and help deal with issues such as changing foliage in image registration, which classical handcrafted features are unable to address. These features are extracted from a semantic segmentation network and have shown good robustness and accuracy in registering aerial images across years and seasons in the experiments.
NimbleReg: A light-weight deep-learning framework for diffeomorphic image registration
Hawkes Institute \& Department of Computer Science, University College London, London, UK AINOSTICS ltd., Manchester, UK ESIR, Universit\'e de Rennes, Rennes, France
Research institute for signals, systems and computational intelligence, sinc(i), FICH-UNL/CONICET（信号、系统与计算智能研究所 sinc(i)，FICH-UNL/CONICET）；University of Oxford（牛津大学）；TheraPanacea；CentraleSupelec（中央理工-高等电力学院）；Universite Paris-Saclay（巴黎-萨克雷大学）
Deformable registration has been one of the pillars of biomedical image computing. Conventional approaches refer to the definition of a similarity criterion that, once endowed with a deformation model and a smoothness constraint, determines the optimal transformation to align two given images. The definition of this metric function is among the most critical aspects of the registration process. We argue that incorporating semantic information (in the form of anatomical segmentation maps) into the registration process will further improve the accuracy of the results. In this paper, we propose a…
The University of Tokyo（东京大学）；Simon Fraser University（西蒙弗雷泽大学）；University of Alberta（阿尔伯塔大学）
Probabilistic image registration methods estimate the posterior distribution of transformation. The conventional way of interpreting the transformation posterior is to use the mode as the most likely transformation and assign its corresponding intensity to the registered voxel. Meanwhile, summary statistics of the posterior are employed to evaluate the registration uncertainty, that is the trustworthiness of the registered image. Despite the wide acceptance, this convention has never been justified. In this paper, based on illustrative examples, we question the correctness and usefulness of co…
UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）
We present a method to predict image deformations based on patch-wise image appearance. Specifically, we design a patch-based deep encoder-decoder network which learns the pixel/voxel-wise mapping between image appearance and registration parameters. Our approach can predict general deformation parameterizations, however, we focus on the large deformation diffeomorphic metric mapping (LDDMM) registration model. By predicting the LDDMM momentum-parameterization we retain the desirable theoretical properties of LDDMM, while reducing computation time by orders of magnitude: combined with patch pr…
In current biological and medical research, statistical shape modeling (SSM) provides an essential framework for the characterization of anatomy/morphology. Such analysis is often driven by the identification of a relatively small number of geometrically consistent features found across the samples of a population. These features can subsequently provide information about the population shape variation. Dense correspondence models can provide ease of computation and yield an interpretable low-dimensional shape descriptor when followed by dimensionality reduction. However, automatic methods for…
In this paper, a method for Automatic Image Registration (AIR) through histogram is proposed. Automatic image registration is one of the crucial steps in the analysis of remotely sensed data. A new acquired image must be transformed, using image registration techniques, to match the orientation and scale of previous related images. This new approach combines several segmentations of the pair of images to be registered. A relaxation parameter on the histogram modes delineation is introduced. It is followed by characterization of the extracted objects through the objects area, axis ratio, and pe…
Deformable templates are essential to large-scale medical image registration, segmentation, and population analysis. Current conventional and deep network-based methods for template construction use only regularized registration objectives and often yield templates with blurry and/or anatomically implausible appearance, confounding downstream biomedical interpretation. We reformulate deformable registration and conditional template estimation as an adversarial game wherein we encourage realism in the moved templates with a generative adversarial registration framework conditioned on flexible i…
