Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration
University College London（伦敦大学学院）；University of Oxford（牛津大学）
Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M. Moore, Mark Emberton, Sébastien Ourselin, … 展开作者
Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M. Moore, Mark Emberton, Sébastien Ourselin, J. Alison Noble, Dean C. Barratt, Tom Vercauteren
One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.
Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets…
Tsinghua University（清华大学）；Shenzhen International Graduate School（深圳国际研究生院）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根妇女医院）
The loss function of an unsupervised multimodal image registration framework has two terms, i.e., a metric for similarity measure and regularization. In the deep learning era, researchers proposed many approaches to automatically learn the similarity metric, which has been shown effective in improving registration performance. However, for the regularization term, most existing multimodal registration approaches still use a hand-crafted formula to impose artificial properties on the estimated deformation field. In this work, we propose a unimodal cyclic regularization training pipeline, which …
Institute of Medical Informatics, Universität zu Lübeck（吕贝克大学医学信息学研究所）
Multimodal image registration is a very challenging problem for deep learning approaches. Most current work focuses on either supervised learning that requires labelled training scans and may yield models that bias towards annotated structures or unsupervised approaches that are based on hand-crafted similarity metrics and may therefore not outperform their classical non-trained counterparts. We believe that unsupervised domain adaptation can be beneficial in overcoming the current limitations for multimodal registration, where good metrics are hard to define. Domain adaptation has so far been…
multiGradICON: A Foundation Model for Multimodal Medical Image Registration
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Brigham and Women’s Hospital（布莱根妇女医院）；ÉTS Montréal（蒙特利尔工程学院）；Boston University（波士顿大学）；Kitware Inc.（Kitware公司）
We describe a point-set registration algorithm based on a novel free point transformer (FPT) network, designed for points extracted from multimodal biomedical images for registration tasks, such as those frequently encountered in ultrasound-guided interventional procedures. FPT is constructed with a global feature extractor which accepts unordered source and target point-sets of variable size. The extracted features are conditioned by a shared multilayer perceptron point transformer module to predict a displacement vector for each source point, transforming it into the target space. The point …
Southwest Jiaotong University（西南交通大学）；Xinyang Normal University（信阳师范大学）
Over the past few decades, with the rapid development of global aerospace and aerial remote sensing technology, the types of sensors have evolved from the traditional monomodal sensors (e.g., optical sensors) to the new generation of multimodal sensors [e.g., multispectral, hyperspectral, light detection and ranging (LiDAR) and synthetic aperture radar (SAR) sensors]. These advanced devices can dynamically provide various and abundant multimodal remote sensing images with different spatial, temporal, and spectral resolutions according to different application requirements. Since then, it is of…
Multispectral and multimodal images are of important usage in the field of multi-source visual information fusion. Due to the alternation or movement of image devices, the acquired multispectral and multimodal images are usually misaligned, and hence image registration is pre-requisite. Different from the registration of common images, the registration of multispectral or multimodal images is a challenging problem due to the nonlinear variation of intensity and gradient. To cope with this challenge, we propose the phase congruency network (PCNet) to enhance the structure similarity of multispe…
We present Free Point Transformer (FPT) - a deep neural network architecture for non-rigid point-set registration. Consisting of two modules, a global feature extraction module and a point transformation module, FPT does not assume explicit constraints based on point vicinity, thereby overcoming a common requirement of previous learning-based point-set registration methods. FPT is designed to accept unordered and unstructured point-sets with a variable number of points and uses a "model-free" approach without heuristic constraints. Training FPT is flexible and involves minimizing an intuitive …
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）
Deformable image registration between Computed Tomography (CT) images and Magnetic Resonance (MR) imaging is essential for many image-guided therapies. In this paper, we propose a novel translation-based unsupervised deformable image registration method. Distinct from other translation-based methods that attempt to convert the multimodal problem (e.g., CT-to-MR) into a unimodal problem (e.g., MR-to-MR) via image-to-image translation, our method leverages the deformation fields estimated from both: (i) the translated MR image and (ii) the original CT image in a dual-stream fashion, and automati…
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Whirlpool（惠而浦）
We present WSSAMNet, a weakly supervised method for medical image registration. Ours is a two step method, with the first step being the computation of segmentation masks of the fixed and moving volumes. These masks are then used to attend to the input volume, which are then provided as inputs to a registration network in the second step. The registration network computes the deformation field to perform the alignment between the fixed and the moving volumes. We study the effectiveness of our technique on the BraTSReg challenge data against ANTs and VoxelMorph, where we demonstrate that our me…
Southwest Jiaotong University（西南交通大学）；University of Trento（特伦托大学）；Purdue University（普渡大学）
Automatic registration of multimodal remote sensing data (e.g., optical, LiDAR, SAR) is a challenging task due to the significant non-linear radiometric differences between these data. To address this problem, this paper proposes a novel feature descriptor named the Histogram of Orientated Phase Congruency (HOPC), which is based on the structural properties of images. Furthermore, a similarity metric named HOPCncc is defined, which uses the normalized correlation coefficient (NCC) of the HOPC descriptors for multimodal registration. In the definition of the proposed similarity metric, we first…
Tsinghua University（清华大学）；Shenzhen International Graduate School（深圳国际研究生院）；Harvard Medical School（哈佛医学院）；Brigham and Women’s Hospital（布莱根和妇女医院）
Multimodal image registration (MIR) is a fundamental procedure in many image-guided therapies. Recently, unsupervised learning-based methods have demonstrated promising performance over accuracy and efficiency in deformable image registration. However, the estimated deformation fields of the existing methods fully rely on the to-be-registered image pair. It is difficult for the networks to be aware of the mismatched boundaries, resulting in unsatisfactory organ boundary alignment. In this paper, we propose a novel multimodal registration framework, which leverages the deformation fields estima…
Southwest Jiaotong University（西南交通大学）；Beijing Institute of Technology（北京理工学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing（测绘遥感信息工程国家重点实验室）
Co-registration of multimodal remote sensing images is still an ongoing challenge because of nonlinear radiometric differences (NRD) and significant geometric distortions (e.g., scale and rotation changes) between these images. In this paper, a robust matching method based on the Steerable filters is proposed consisting of two critical steps. First, to address severe NRD, a novel structural descriptor named the Steerable Filters of first- and second-Order Channels (SFOC) is constructed, which combines the first- and second-order gradient information by using the steerable filters with a multi-…
Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration
IMAG2, Institut IMAGINE, Université Paris Cité（IMAG2，IMAGINE研究所，巴黎西岱大学）；Replico SAS（Replico公司）；Université Paris Cité, Department of Pediatric Surgery, Hôpital Necker Enfants-Malades, Assistance Publique–Hôpitaux de Paris (AP-HP)（巴黎西岱大学，内克尔儿童医院小儿外科，巴黎公立医院集团）；Sorbonne Université, CNRS, LIP6（索邦大学，法国国家科学研究中心，LIP6）；LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎电信，巴黎综合理工学院）
Unsupervised MR-US Multimodal Image Registration with Multilevel Correlation Pyramidal Optimization
School of Artificial Intelligence and Robotics, Hunan University, Changsha, Hunan, China（人工智能与机器人学院，湖南大学，长沙，湖南，中国）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, Hunan, China（机器人视觉感知与控制技术国家工程研究中心，湖南大学，长沙，湖南，中国）；National University of Singapore（新加坡国立大学）；Cornell University（康奈尔大学）
Multimodal Remote Sensing Image Registration with Accuracy Estimation at Local and Global Scales
National Aerospace University（国立航空航天大学）；University of Rennes 1（雷恩第一大学）
本文主要研究遥感图像配准的潜在精度问题。我们研究如何在没有地面实况的情况下估计这种精度，并用于提高单模态和多模态图像对的配准质量。在局部尺度的图像片段，Cramer-Rao下界（CRLB）的配准误差估计的每个局部对应之间的粗配准对的图像。该CRLB由局部图像纹理和噪声属性定义。与仅在配准过程的输出处评估配准精度的标准方法相反，我们将这些有价值的信息用作额外的输入知识。这对发现和剔除异常值以及改进几何变换模型参数的估计有很大帮助。基于这些思想，提出了一种新的基于区域的配准方法RAE（Registration with Accuracy Estimation）。除了其以高精度自动配准非常复杂的多模态图像对的能力之外，RAE方法在全局尺度上提供配准精度作为几何变换模型参数的估计误差的协方差矩阵或作为逐点配准标准偏差。这种精度不依赖于任何地面实况可用性，并单独表征每对配准图像。因此，RAE方法可以识别保证预定义配准精度的图像区域。RAE方法被证明是成功的，达到亚像素精度，同时注册8个复杂的单/多模态和多时相图像对，包括光学光学，光学雷达，光学数字高程模型（DEM）图像和DEM雷达的情况下。在比较中采用的其他方法不能以稳定的方式提供相同测试用例的准确结果。
University College London（伦敦大学学院）；Verdure Imaging, Inc.（Verdure成像公司）；Queen’s University（女王大学）
Free Point Transformer (FPT) has been proposed as a data-driven, non-rigid point set registration approach using deep neural networks. As FPT does not assume constraints based on point vicinity or correspondence, it may be trained simply and in a flexible manner by minimizing an unsupervised loss based on the Chamfer Distance. This makes FPT amenable to real-world medical imaging applications where ground-truth deformations may be infeasible to obtain, or in scenarios where only a varying degree of completeness in the point sets to be aligned is available. To test the limit of the corresponden…
Multimodal Deformable Image Registration for Long-COVID Analysis Based on Progressive Alignment and Multi-perspective Loss
University of Oxford（牛津大学）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）；Big Data Institute, University of Oxford（牛津大学大数据研究所）；Oxford University Hospitals NHS Foundation Trust（牛津大学医院 NHS 基金会信托基金）；Oxford Radiology Research Unit（牛津放射学研究单元）
Weakly supervised alignment and registration of MR-CT for cervical cancer radiotherapy
College of Medicine and Biological Information Engineering（医学与生物信息工程学院）；Cancer Hospital of Dalian University of Technology（大连理工大学肿瘤医院）；Cancer Hospital of China Medical University（中国医科大学附属肿瘤医院）
Disentangle-then-Align: Non-Iterative Hybrid Multimodal Image Registration via Cross-Scale Feature Disentanglement
Faculty of Engineering and IT, University of Technology Sydney（新南威尔士大学工程与信息技术学院）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection
School of Electronics and Information Engineering, Wuxi University（无锡大学电子与信息工程学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）
Weakly-Supervised Learning of Metric Aggregations for Deformable Image Registration
Research institute for signals, systems and computational intelligence, sinc(i), FICH-UNL/CONICET（信号、系统与计算智能研究所 sinc(i)，FICH-UNL/CONICET）；University of Oxford（牛津大学）；TheraPanacea；CentraleSupelec（中央理工-高等电力学院）；Universite Paris-Saclay（巴黎-萨克雷大学）
Southwest Jiaotong University（西南交通大学）；University of Trento（特伦托大学）；Purdue University（普渡大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
While image registration has been studied in remote sensing community for decades, registering multimodal data [e.g., optical, LiDAR, SAR, and map] remains a challenging problem because of significant nonlinear intensity differences between such data. To address this problem, this paper presents a fast and robust matching framework integrating local descriptors for multimodal registration. In the proposed framework, a local descriptor, such as Histogram of Oriented Gradient (HOG), Local Self Similarity (LSS), or Speeded-Up Robust Feature (SURF), is first extracted at each pixel to form a pixel…
A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy
Department of Radiation Oncology, Mayo Clinic, Phoenix, Arizona, USA（梅奥诊所放射肿瘤科）；School of Biomedical Engineering, ShanghaiTech University, Shanghai, China（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd., Shanghai 200230, China（上海联合影像智能科技有限公司）；Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China（上海人工智能实验室）；Shanghai Clinical Research and Trial Center, Shanghai 201210, China（上海临床研究与试验中心）
The Brain Resection Multimodal Image Registration (ReMIND2Reg) 2025 Challenge
Inria Saclay Île-de-France（法国里昂萨克利研究所）；CEA（法国原子能委员会）；Université Paris-Saclay（巴黎-萨克利大学）；Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎脑研究所）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化技术研究所）；Inserm（法国国家医学研究院）；AP-HP（法国国家医院集团）；Hôpital de la Pitié Salpêtrière（皮蒂埃-萨尔普里埃尔医院）；Harvard Medical School（哈佛医学院）；Brigham and Women's Hospital（布里根医院）；Department of Radiology and Radiological Science（放射学与放射科学系）；Johns Hopkins Medical School（约翰霍普金斯医学院）；Institute of Medical Informatics（医学信息学研究所）；University of Lübeck（吕贝克大学）；Image Analysis and Communications Laboratory（图像分析与通信实验室）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Johns Hopkins University（约翰霍普金斯大学）；CSAIL（媒体实验室）
Image registration is a fundamental building block for various applications in medical image analysis. To better explore the correlation between the fixed and moving images and improve registration performance, we propose a novel deep learning network, Co-Attention guided Registration Network (CAR-Net). CAR-Net employs a co-attention block to learn a new representation of the inputs, which drives the registration of the fixed and moving images. Experiments on UK Biobank cardiac cine-magnetic resonance image data demonstrate that CAR-Net obtains higher registration accuracy and smoother deforma…
University of Magdeburg（马格德堡大学）；University Hospital Magdeburg（马格德堡大学医院）
Recent studies demonstrated the eligibility of convolutional neural networks (CNNs) for solving the image registration problem. CNNs enable faster transformation estimation and greater generalization capability needed for better support during medical interventions. Conventional fully-supervised training requires a lot of high-quality ground truth data such as voxel-to-voxel transformations, which typically are attained in a too tedious and error-prone manner. In our work, we use weakly-supervised learning, which optimizes the model indirectly only via segmentation masks that are a more access…
Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network
Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University（遥感图像处理与融合组，电子工程学院，西安电子科技大学）；National Key Laboratory of Radar Signal Processing, Xidian University（雷达信号处理国家级实验室，西安电子科技大学）
Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity
Northwestern Polytechnical University（西北工业大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能有限公司）；Shanghai Jiao Tong University（上海交通大学）
ShanghaiTech University（上海科技大学）；University of Zürich（苏黎世大学）；ETH Zürich（苏黎世联邦理工学院）
Multi-spectral optoacoustic tomography (MSOT) is an emerging optical imaging method providing multiplex molecular and functional information from the rodent brain. It can be greatly augmented by magnetic resonance imaging (MRI) that offers excellent soft-tissue contrast and high-resolution brain anatomy. Nevertheless, registration of multi-modal images remains challenging, chiefly due to the entirely different image contrast rendered by these modalities. Previously reported registration algorithms mostly relied on manual user-dependent brain segmentation, which compromised data interpretation …
Image registration aims to establish spatial correspondence across pairs, or groups of images, and is a cornerstone of medical image computing and computer-assisted-interventions. Currently, most deep learning-based registration methods assume that the desired deformation fields are globally smooth and continuous, which is not always valid for real-world scenarios, especially in medical image registration (e.g. cardiac imaging and abdominal imaging). Such a global constraint can lead to artefacts and increased errors at discontinuous tissue interfaces. To tackle this issue, we propose a weakly…
In recent years, we have witnessed a surge of interests in learning a suitable distance metric from weakly supervised data. Most existing methods aim to pull all the similar samples closer while push the dissimilar ones as far as possible. However, when some classes of the dataset exhibit multimodal distribution, these goals conflict and thus can hardly be concurrently satisfied. Additionally, to ensure a valid metric, many methods require a repeated eigenvalue decomposition process, which is expensive and numerically unstable. Therefore, how to learn an appropriate distance metric from weakly…
Tsinghua University（清华大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Deformable image registration (DIR) is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in DIR. Most deep learning approaches use the so-called mono-stream "high-to-low, low-to-high" network structure, and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of de…
