While thermal optical registered datasets are becoming widely available, most of these works are based on image pairs which are pre-registered. However, thermal imagers where these images are registered by default are quite expensive. We present in this work, a thermal image registration technique which is computationally lightweight, and can be employed regardless of the resolution of the images captured. We use 2 different thermal imagers to create a completely new database and introduce it as a part of this work as well. The images captured are based on 5 different classes and encompass subjects like the Prayagraj Kumbh Mela, one of the largest public fairs in the world, captured over a period of 2 years.
University of Sao Paulo（圣保罗大学）；Faculty of Philosophy, Science and Languages of Ribeirao Preto（里贝朗普雷图哲学、科学与语言学院）；Department of Physics Applied to Medicine and Biology（医学与生物学应用物理系）
Mutual information (MI) is the standard method used in image registration and the most studied one but can diverge and produce wrong results when used in an automated manner. In this study we compared the results of the ITK Mattes MI function, used in 3D Slicer and ITK derived software solutions, and our own MICUDA Shannon and Tsallis MI functions under the translation, rotation and scale transforms in a 3D mathematical space. This comparison allows to understand why registration fails in some circumstances and how to produce a more robust automated algorithm to register medical images. Since …
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）
Many applications in image-guided surgery and therapy require fast and reliable non-linear, multi-modal image registration. Recently proposed unsupervised deep learning-based registration methods have demonstrated superior performance compared to iterative methods in just a fraction of the time. Most of the learning-based methods have focused on mono-modal image registration. The extension to multi-modal registration depends on the use of an appropriate similarity function, such as the mutual information (MI). We propose guiding the training of a deep learning-based registration method with MI…
Image registration is a key operation in medical image processing, allowing a plethora of applications. Mutual information (MI) is consolidated as a robust similarity metric often used for medical image registration. Although MI provides a robust medical image registration, it usually fails when the needed image transform is too big due to MI local maxima traps. In this paper, we propose and evaluate a generalized parametric MI as an affine registration cost function. We assessed the generalized MI (GMI) functions for separable affine transforms and exhaustively evaluated the GMI mathematical …
Institute for Medical Imaging Technology, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200030, China（中国上海200030，上海交通大学生物医学工程学院医学影像技术研究所）；Department of Radiology and Biomedical Research Imaging Center (BRIC), University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA（美国北卡罗来纳州教堂山27599，北卡罗来纳大学教堂山分校放射学与生物医学研究成像中心（BRIC））；North China Electric Power University, Beijing, China（中国北京华北电力大学）；Shanghai United Imaging Intelligence Co., Ltd, Shanghai, China（中国上海上海联影智能科技有限公司）；Shanghai Cancer Center, Fudan University, Shanghai, China（中国上海复旦大学附属肿瘤医院）
Thermal ablation is a minimally invasive procedure for treat-ing small or unresectable tumors. Although CT is widely used for guiding ablation procedures, the contrast of tumors against surrounding normal tissues in CT images is often poor, aggravating the difficulty in accurate thermal ablation. In this paper, we propose a fast MR-CT image registration method to overlay a pre-procedural MR (pMR) image onto an intra-procedural CT (iCT) image for guiding the thermal ablation of liver tumors. By first using a Cycle-GAN model with mutual information constraint to generate synthesized CT (sCT) ima…
University of Nevada, Reno（内华达大学雷诺分校）；NTNU - Norwegian University of Science and Technology（挪威科技大学）
This report accompanies a dataset release on visual and thermal camera data and details a procedure followed to align such multi-modal camera frames in order to provide pixel-level correspondence between the two without using intrinsic or extrinsic calibration information. To achieve this goal we benefit from progress in the domain of multi-modal image alignment and specifically employ the Mattes Mutual Information Metric to guide the registration process. In the released dataset we release both the raw visual and thermal camera data, as well as the aligned frames, alongside calibration parame…
Quantib-U（Quantib-U公司）；Amsterdam University Medical Center（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；University Medical Center Utrecht（乌得勒支大学医学中心）
Deep neural networks are increasingly used for pair-wise image registration. We propose to extend current learning-based image registration to allow simultaneous registration of multiple images. To achieve this, we build upon the pair-wise variational and diffeomorphic VoxelMorph approach and present a general mathematical framework that enables both registration of multiple images to their geodesic average and registration in which any of the available images can be used as a fixed image. In addition, we provide a likelihood based on normalized mutual information, a well-known image similarit…
This work presents a new method for unsupervised thermal image classification and semantic segmentation by transferring knowledge from the RGB domain using a multi-domain attention network. Our method does not require any thermal annotations or co-registered RGB-thermal pairs, enabling robots to perform visual tasks at night and in adverse weather conditions without incurring additional costs of data labeling and registration. Current unsupervised domain adaptation methods look to align global images or features across domains. However, when the domain shift is significantly larger for cross-m…
Uppsala University（乌普萨拉大学）；University of Copenhagen（哥本哈根大学）
Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration…
Search-MIND: Training-Free Multi-Modal Medical Image Registration
Lab for Uncertainty in Data and Decision Making (LUCID), Nottingham Biomedical Research Centre (BRC), School of Medicine, University of Nottingham, UK（英国诺丁汉大学医学院诺丁汉生物医学研究中心（BRC）数据与决策不确定性实验室（LUCID））
Center for Image Sciences, University Medical Center Utrecht, the Netherlands（荷兰乌得勒支大学医学中心影像科学中心）；Department of Radiology, University Medical Center Utrecht, the Netherlands（荷兰乌得勒支大学医学中心放射科）
Motion correction of dynamic contrast enhanced magnetic resonance images (DCE-MRI) is a challenging task, due to changes in image appearance. In this study a groupwise registration, using a principle component analysis (PCA) based metric,1 is evaluated for clinical DCE MRI of the liver. The groupwise registration transforms the images to a common space, rather than to a reference volume as conventional pairwise methods do, and computes the similarity metric on all volumes simultaneously. This groupwise registration method is compared to a pairwise approach using a mutual information metric. Cl…
Deep Learning in Medical Image Registration: Magic or Mirage?
Computer and Information Science（计算机与信息科学学院）；Electrical and Systems Engineering（电气与系统工程学院）；Radiology（放射学系）；Penn Image Computing and Science Laboratory（宾夕法尼亚图像计算与科学实验室）
University of Sussex（萨塞克斯大学）；Polytechnique Montreal（蒙特利尔理工学院）；Cardiff University（卡迪夫大学）；University College London（伦敦大学学院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Nonlinear inter-modality registration is often challenging due to the lack of objective functions that are good proxies for alignment. Here we propose a synthesis-by-registration method to convert this problem into an easier intra-modality task. We introduce a registration loss for weakly supervised image translation between domains that does not require perfectly aligned training data. This loss capitalises on a registration U-Net with frozen weights, to drive a synthesis CNN towards the desired translation. We complement this loss with a structure preserving constraint based on contrastive l…
Applications based on synergistic integration of optical imagery and LiDAR data are receiving a growing interest from the remote sensing community. However, a misaligned integration between these datasets may fail to fully profit the potential of both sensors. In this regard, an optimum fusion of optical imagery and LiDAR data requires an accurate registration. This is a complex problem since a versatile solution is still missing, especially when considering the context where data are collected at different times, from different platforms, under different acquisition configurations. This paper…
University of Tennessee（田纳西大学）；Applied Research LLC（应用研究有限责任公司）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）
Hyperspectral images (HSI) provide rich spectral information that contributed to the successful performance improvement of numerous computer vision tasks. However, it can only be achieved at the expense of images' spatial resolution. Hyperspectral image super-resolution (HSI-SR) addresses this problem by fusing low resolution (LR) HSI with multispectral image (MSI) carrying much higher spatial resolution (HR). All existing HSI-SR approaches require the LR HSI and HR MSI to be well registered and the reconstruction accuracy of the HR HSI relies heavily on the registration accuracy of different …
Images from Positron Emission Tomography (PET) deliver functional data such as perfusion and metabolism. On the other hand, images from Magnetic Resonance Imaging (MRI) provide information describing anatomical structures. Fusing the complementary information from the two modalities is helpful in oncology. In this project, we implemented a complete tool allowing semi-automatic MRI-PET registration for small animal imaging in the preclinical studies. A two-stage hierarchical registration approach is proposed. First, a global affine registration is applied. For robust and fast registration, prin…
Ryerson University（瑞尔森大学）；University of Toronto（多伦多大学）；Institute for Biomedical Engineering, Science and Technology (iBEST)（生物医学工程、科学与技术研究所（iBEST））
We propose FlowReg, a deep learning-based framework for unsupervised image registration for neuroimaging applications. The system is composed of two architectures that are trained sequentially: FlowReg-A which affinely corrects for gross differences between moving and fixed volumes in 3D followed by FlowReg-O which performs pixel-wise deformations on a slice-by-slice basis for fine tuning in 2D. The affine network regresses the 3D affine matrix based on a correlation loss function that enforces global similarity. The deformable network operates on 2D image slices based on the optical flow netw…
University of Washington（华盛顿大学）；Magic Leap Inc.（Magic Leap公司）
Retinal template matching and registration is an important challenge in teleophthalmology with low-cost imaging devices. However, the images from such devices generally have a small field of view (FOV) and image quality degradations, making matching difficult. In this work, we develop an efficient and accurate retinal matching technique that combines dimension reduction and mutual information (MI), called RetinaMatch. The dimension reduction initializes the MI optimization as a coarse localization process, which narrows the optimization domain and avoids local optima. The effectiveness of Reti…
Deep Regression 2D-3D Ultrasound Registration for Liver Motion Correction in Focal Tumor Thermal Ablation
Robarts Research Institute（罗伯茨研究所）；Western University（西安大略大学）；School of Biomedical Engineering, Western University（西安大略大学生物医学工程学院）；Lawson Health Research Institute（劳森健康研究所）
Accurate and robust whole heart substructure segmentation is crucial in developing clinical applications, such as computer-aided diagnosis and computer-aided surgery. However, segmentation of different heart substructures is challenging because of inadequate edge or boundary information, the complexity of the background and texture, and the diversity in different substructures' sizes and shapes. This article proposes a framework for multi-class whole heart segmentation employing non-rigid registration-based probabilistic atlas incorporating the Bayesian framework. We also propose a non-rigid r…
