arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03341cs.CV
收藏

基于结构化状态空间对偶性的跨模态特征融合多模态图像配准网络

Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network

Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University(遥感图像处理与融合组,电子工程学院,西安电子科技大学);National Key Laboratory of Radar Signal Processing, Xidian University(雷达信号处理国家级实验室,西安电子科技大学)

Zhikang Li, Yan Wu, Xin Hu, Yi Dai, Ming Li

AI总结:

提出RegNetMamba-2算法,利用结构化状态空间对偶性(SSD)在粗到细匹配过程中提取局部和全局结构特征,通过跨模态交互和多尺度融合模块实现多模态图像配准,在多个数据集上取得高效性能。

摘要:

在多模态图像配准中,主要挑战在于共享结构信息的提取。与Transformer相比,结构化状态空间对偶性(SSD)在训练和推理过程中能以更高效率提取更全面的全局结构特征。受这些优势启发,我们提出了一种新的多模态图像配准算法,命名为RegNetMamba-2。我们的算法将SSD融入粗到细的匹配过程中,以有效提取局部和全局结构特征。首先,在网络中应用SSD于三种不同尺度进行多模态特征提取。为了增强局部表示,我们通过SSD的特征缩放函数更加关注前景边缘和结构信息。其次,针对输入图像的共享特征提取和所有尺度的多模态特征融合,我们提出了基于SSD的跨模态特征融合模型,包括跨模态特征交互(CMI)模块和多尺度特征融合(MSF)模块。CMI模块通过交叉形式的SSD用于每个尺度的跨模态特征提取。MSF模块旨在采用渐进式向上融合方式在特征层面获取精细特征,包含所有尺度的多模态特征。遵循粗到细策略,收集来自CMI的1/8尺度特征和来自MSF的1/2尺度特征以计算匹配概率分数。然后我们通过像素级对应关系分别建立匹配过程。大量实验表明,与最先进的基于深度学习的算法相比,RegNetMamba-2在以下数据集上的多模态图像配准性能和效率均取得了良好效果:VIS-SAR(OSDataset)、VIS-IR(LGHD/RoadSence)和VIS-NIR(RGB-NIR sense)。

英文摘要:

In multi-modal image registration, the primary challenge lies in shared structural information extraction. Compared to Transformers, Structured State Space Duality (SSD) offers greater global structural feature extraction with higher efficiency during training and inference. Inspired by these advantages, we propose a novel algorithm for multi-modal image registration, named RegNetMamba-2. Our algorithm incorporates SSD into coarse-to-fine matching process to extract local and global structural features effectively. Firstly, SSD is applied in three different scales for multi-modal feature extraction in our network. To strengthen local representation, we pay more attention on foreground edge and structural information by feature scaling function of SSD. Secondly, for shared feature extraction of input images and multi-modal feature fusion in all scales, we propose cross-modality feature fusion model based on SSD, consisting of Cross-Modality feature Interaction (CMI) module and Multi-Scale feature Fusion (MSF) module. CMI module is designed for cross-modality feature extraction of each scale by SSD in cross form. MSF module is designed to employ a progressive upward fusion in feature-level to obtain fine features, consisting of multi-modal features in all scales. Following coarse-to-fine, the features in 1/8 scale from CMI and 1/2 scale from MSF are collected to calculate matching probability scores. Then we respectively establish matching process by correspondences of pixel-wise. Extensive experiments demonstrate that comparing with state-of-the-art deep-learning based algorithms, RegNetMamba-2 has achieved good effects in both performance and efficiency for multi-modal image registration on the following datasets: VIS-SAR (OSDataset), VIS-IR (LGHD/RoadSence) and VIS-NIR (RGB-NIR sense).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑